From 43958fd26a2a9d3cb31f138712f48dd0c3baedb9 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 7 Sep 2026 16:38:17 -0400 Subject: [PATCH 1/3] feat(starlark,on-demand): the third-party fixes worth taking, plus an MQTT bridge Analysis of ant456/ledmatrix-fixes-repo, a third-party collection of patches and services built while running this project on Starlark apps under MQTT control. Its patches are whole-file copies taken against an older tree, so applying them as written would revert #523's frame pacing, #534's display() bool returns and the GitHub token masking in plugins_manager.js. Three of its claimed fixes are already in main, and its api_v3 Starlark routes are #535's. What follows is the rest -- verified against current code, and reimplemented where the patch's approach did not hold up. **On-demand display.** `pinned` reached the controller from the API, was stored on it and republished in the status payload, but never narrowed the rotation -- a pinned request still cycled every mode its plugin owns. Right for a sports plugin, whose modes are views of one subject; wrong for a plugin whose modes are unrelated, which is every Starlark app. Now honoured, and it survives a restart. Restarting while on-demand was active loaded *only* the on-demand plugin, so normal rotation had nothing to return to for the life of the process -- and a restart mid-session is routine, since that is how an update is applied. The panel came back cycling one plugin's modes with no way out but clearing the cache by hand. Every enabled plugin loads now; on-demand still resumes on its saved mode. Stop requests are exempt from the duplicate guards on purpose, so that a second click stops a mode a race left running -- which means consuming the mailbox is the only thing that ends one. It was never consumed, so the same stop was re-read and re-processed on every poll, forever. Both paths now share one compare-before-delete helper. **Starlark rendering.** `extract_schema` parsed the source with a regex, which can only see option lists written out literally: an app whose dropdown is filled from a live API call inside `get_schema()` came back empty, and the config form offered nothing to pick. Now runs `pixlet schema`, which executes the app, and falls back to the parser when Pixlet is absent, too old for the subcommand, or the app fails to run. The third-party patch replaced the parser outright and hardcoded /usr/local/bin/pixlet; this keeps the fallback and the binary search. A `|` in a config value was dropped by a shell-metacharacter filter, though the command is a list with no shell involved -- and apps do use it as a separator inside one value. The key went missing silently and the app rendered its own "not configured" screen with nothing to say why. And a 0-byte render was reported as success: Pixlet exits 0 and writes nothing when an app has no content, which read downstream as a working app drawing a black panel. **Starlark display.** `display()` ignored the mode it was called with, so a specific app could not be addressed. It now accepts `display_mode` -- which is the whole mechanism, since the controller inspects the signature before passing it. Found while there: `_select_next_app` ran only while `current_app` was unset, so with several apps installed the first was picked once and shown forever while the rest were rendered on schedule and never displayed. And `enable_scrolling` was missing, so multi-frame apps were called once per rotation slot and never advanced past frame one. **GET /api/v3/display/modes.** Every mode that can be requested on-demand, with the plugin that owns it. Nothing exposed this, so anything driving the display from outside the web UI read each plugin's manifest.json off disk and reimplemented PluginManager's fallbacks. It also triggers discovery, which is otherwise lazy and normally happens because a person opened the dashboard. **integrations/mqtt_bridge.** Home Assistant control over MQTT Discovery: a mode select, a stop button, power, brightness. Rewritten against the API rather than the filesystem, so it needs no read access to config.json and cannot drift from the web UI. paho-mqtt 2.x VERSION2, TLS, an availability topic that is also the last will, and secrets from the environment. **Two opt-in extras.** A DNS single-request unit, for glibc's parallel A/AAAA lookup stalling ~5s per name on routers that answer only the A query -- which makes any plugin calling an external API slow and Starlark apps, which have a render timeout, fail outright. And a Pixlet config editor: a script you run and Ctrl+C rather than the third-party version's always-on unauthenticated Flask service, since it stops the display for the length of a session. Neither is installed by default. Long Starlark app names now wrap instead of overflowing their card. 115 new tests across 5 files. Also unblocked test_starlark_display_contract.py, which was silently skipping wherever fcntl is absent. Whole suite: no new failures against main. Co-Authored-By: Claude Opus 5 --- docs/REST_API_REFERENCE.md | 52 +- integrations/mqtt_bridge/.gitignore | 1 + integrations/mqtt_bridge/README.md | 112 ++++ .../mqtt_bridge/bridge_config.example.json | 13 + .../mqtt_bridge/ledmatrix_mqtt_bridge.py | 530 ++++++++++++++++++ integrations/mqtt_bridge/requirements.txt | 2 + plugin-repos/starlark-apps/manager.py | 29 +- plugin-repos/starlark-apps/pixlet_renderer.py | 108 +++- scripts/install/install_dns_fix.sh | 59 ++ scripts/install/install_mqtt_bridge.sh | 64 +++ scripts/utils/README.md | 28 + scripts/utils/apply_dns_single_request.sh | 69 +++ scripts/utils/pixlet_config_editor.sh | 148 +++++ src/display_controller.py | 276 +++++---- systemd/README.md | 23 + systemd/ledmatrix-dns-fix.service | 16 + systemd/ledmatrix-mqtt-bridge.service | 18 + test/test_api_v3_display_modes.py | 104 ++++ test/test_mqtt_bridge.py | 266 +++++++++ test/test_on_demand_pinning_and_restart.py | 208 +++++++ test/test_pixlet_renderer_contract.py | 210 +++++++ test/test_starlark_display_contract.py | 125 ++++- web_interface/blueprints/api_v3.py | 58 ++ web_interface/static/v3/plugins_manager.js | 5 +- 24 files changed, 2386 insertions(+), 138 deletions(-) create mode 100644 integrations/mqtt_bridge/.gitignore create mode 100644 integrations/mqtt_bridge/README.md create mode 100644 integrations/mqtt_bridge/bridge_config.example.json create mode 100644 integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py create mode 100644 integrations/mqtt_bridge/requirements.txt create mode 100644 scripts/install/install_dns_fix.sh create mode 100644 scripts/install/install_mqtt_bridge.sh create mode 100644 scripts/utils/apply_dns_single_request.sh create mode 100644 scripts/utils/pixlet_config_editor.sh create mode 100644 systemd/ledmatrix-dns-fix.service create mode 100644 systemd/ledmatrix-mqtt-bridge.service create mode 100644 test/test_api_v3_display_modes.py create mode 100644 test/test_mqtt_bridge.py create mode 100644 test/test_on_demand_pinning_and_restart.py create mode 100644 test/test_pixlet_renderer_contract.py diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index f95cb84c0..2f3121512 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -33,7 +33,7 @@ All endpoints return JSON responses with a standard format: > The API blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). > SSE stream endpoints (`/api/v3/stream/*`) are defined directly on the -> Flask app at `app.py:799-809`. There are 94 routes total — see +> Flask app at `app.py:799-809`. There are 111 routes total — see > `web_interface/blueprints/api_v3.py` for the canonical list. --- @@ -223,6 +223,56 @@ Get the current display state and preview image. } ``` +### List Display Modes + +**GET** `/api/v3/display/modes` + +Every display mode that can be requested on-demand, with the plugin that owns +it. This is the list the force-display dialog offers. + +Send the reported `plugin_id` alongside `mode` when starting an on-demand +display: `/display/on-demand/start` falls back to `find_plugin_for_mode` when +`plugin_id` is omitted, and that lookup only sees modes declared in a static +manifest — a plugin whose modes are generated (each installed Starlark app is +one) returns 404 there. + +Triggers plugin discovery, which is otherwise lazy — so a caller that never +opens the dashboard still gets the full list. + +**Query Parameters**: +- `include_disabled` (optional): `1` to include modes belonging to disabled + plugins. They are still valid on-demand targets — the controller enables the + plugin for the duration of the request — and are reported with + `"enabled": false`. + +**Response**: +```json +{ + "status": "success", + "data": { + "modes": [ + { + "mode": "nfl_live", + "plugin_id": "football-scoreboard", + "plugin_name": "Football Scoreboard", + "name": "nfl_live", + "enabled": true + }, + { + "mode": "clock-simple", + "plugin_id": "clock-simple", + "plugin_name": "Simple Clock", + "name": "Simple Clock", + "enabled": true + } + ] + } +} +``` + +`name` is a label for a dropdown: a single-mode plugin's own name, or the raw +mode string for a multi-mode plugin, since there is no per-mode name anywhere. + ### On-Demand Display Status **GET** `/api/v3/display/on-demand/status` diff --git a/integrations/mqtt_bridge/.gitignore b/integrations/mqtt_bridge/.gitignore new file mode 100644 index 000000000..72d5ed415 --- /dev/null +++ b/integrations/mqtt_bridge/.gitignore @@ -0,0 +1 @@ +bridge_config.json diff --git a/integrations/mqtt_bridge/README.md b/integrations/mqtt_bridge/README.md new file mode 100644 index 000000000..fed0d564f --- /dev/null +++ b/integrations/mqtt_bridge/README.md @@ -0,0 +1,112 @@ +# Home Assistant MQTT Bridge + +Control the matrix from Home Assistant: force any plugin or mode on demand, +turn the display on and off, and set brightness — as real HA entities, not +hand-written `mqtt.publish` calls. + +The bridge owns no display logic. It subscribes to one command topic and +turns each message into a call against the same `api_v3` routes the web UI +uses, so behaviour lives in one place. It talks to the API over HTTP only — +no filesystem access — so it can run on the Pi or anywhere that can reach +the web interface. + +## What appears in Home Assistant + +On connect the bridge publishes [MQTT Discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery) +config, so the matrix shows up under **Settings → Devices & Services → MQTT** +with no YAML: + +| Entity | Does | +|---|---| +| `select.ledmatrix_display_mode` | Every mode across enabled plugins. Choosing one force-displays it. | +| `button.ledmatrix_stop_display` | Back to normal rotation. | +| `switch.ledmatrix_power` | Starts/stops the display service. | +| `number.ledmatrix_brightness` | 0–100. | + +State is read back from the API every 30 seconds, so the entities also track +changes made from the web UI or an on-demand window expiring on its own. +All four share an availability topic that is the bridge's MQTT last will: +if the bridge dies, HA greys the controls out rather than leaving them +looking live but inert. + +## Raw commands + +For anything the entities do not cover, publish JSON to the command topic +(`ledmatrix/command` by default): + +```jsonc +// Force a mode. plugin_id is optional — the bridge fills it in from +// /api/v3/display/modes. +{"action": "display", "mode": "nfl_live"} + +// duration is seconds; pinned holds this one mode instead of rotating +// through every mode the plugin owns. Pin Starlark apps, where each mode +// is an unrelated widget; leave a sports plugin unpinned so live/recent/ +// upcoming still cycle. +{"action": "display", "plugin_id": "starlark-apps", "mode": "aquarium", + "duration": 300, "pinned": true} + +{"action": "stop_display"} +{"action": "power", "state": "on"} +{"action": "brightness", "value": 75} + +// Re-read the mode list and re-publish discovery, after installing a plugin +{"action": "refresh"} +``` + +Every command publishes its outcome to `/status`, and current +state to `/state`. + +## Requirements + +- A LEDMatrix install with its web interface reachable (default `http://localhost:5000`) +- An MQTT broker that Home Assistant is also connected to +- Python 3 with `paho-mqtt` 2.x and `requests` + +## Install + +```bash +sudo ./scripts/install/install_mqtt_bridge.sh +``` + +That copies `bridge_config.example.json` to `bridge_config.json` on first +run, installs the dependencies, and enables `ledmatrix-mqtt-bridge.service`. +Edit the config with your broker details and re-run it. + +```json +{ + "mqtt_host": "192.168.1.10", + "mqtt_port": 1883, + "mqtt_username": "ledmatrix", + "mqtt_password": null, + "mqtt_topic": "ledmatrix/command", + "mqtt_tls": false, + "ledmatrix_api_base": "http://localhost:5000" +} +``` + +`bridge_config.json` is gitignored. Any key can also be supplied through the +environment as `LEDMATRIX_MQTT_` (`LEDMATRIX_MQTT_MQTT_PASSWORD`, say), +which keeps a broker password out of a file on disk — put it in a systemd +drop-in with `Environment=` or `EnvironmentFile=` instead. + +Set `mqtt_tls: true` for a broker with TLS. `mqtt_tls_insecure` skips +certificate verification and exists only for a self-signed broker on a +trusted LAN; it logs a warning when used. + +To run it in the foreground while setting things up: + +```bash +python3 integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py --config integrations/mqtt_bridge/bridge_config.json +``` + +## Notes + +- Only one thing can be on-demand at a time — the same constraint the web UI has. +- Forcing a mode restarts the display service, so the panel blanks for a moment. +- The mode list comes from `/api/v3/display/modes`, which triggers plugin + discovery itself. Discovery is lazy and normally happens because somebody + opened the dashboard; without that endpoint a bridge that never does would + see an empty list. +- Brightness writes `display.hardware.brightness` through `/api/v3/config/main`. + The display service picks it up on its next restart, not instantly. diff --git a/integrations/mqtt_bridge/bridge_config.example.json b/integrations/mqtt_bridge/bridge_config.example.json new file mode 100644 index 000000000..98188cfc3 --- /dev/null +++ b/integrations/mqtt_bridge/bridge_config.example.json @@ -0,0 +1,13 @@ +{ + "mqtt_host": "192.168.1.10", + "mqtt_port": 1883, + "mqtt_username": "ledmatrix", + "mqtt_password": null, + "mqtt_client_id": "ledmatrix-mqtt-bridge", + "mqtt_topic": "ledmatrix/command", + "mqtt_tls": false, + "ledmatrix_api_base": "http://localhost:5000", + "request_timeout": 15, + "on_demand_duration": null, + "log_level": "INFO" +} diff --git a/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py new file mode 100644 index 000000000..dca657f3d --- /dev/null +++ b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py @@ -0,0 +1,530 @@ +#!/usr/bin/env python3 +"""Control a LEDMatrix display from Home Assistant over MQTT. + +The bridge owns no display logic. It subscribes to one command topic and +turns each message into a call against the same api_v3 routes the web UI +uses, so behaviour stays in one place and this stays a translation layer. + +On connect it publishes Home Assistant MQTT Discovery config, so a matrix +appears in HA as real entities rather than something you drive with +`mqtt.publish` by hand: + + select.ledmatrix_display_mode every mode across enabled plugins; + choosing one force-displays it + button.ledmatrix_stop_display back to normal rotation + switch.ledmatrix_power the display service, on or off + number.ledmatrix_brightness 0-100 + +Anything the entities do not cover is still reachable by publishing JSON +to the command topic: + + {"action": "display", "mode": "nfl_live"} + {"action": "display", "plugin_id": "starlark-apps", "mode": "aquarium", + "duration": 300, "pinned": true} + {"action": "stop_display"} + {"action": "power", "state": "on" | "off"} + {"action": "brightness", "value": 75} + {"action": "refresh"} re-publish discovery after installing a plugin + +Every command publishes its result to /status. + +Run it with `python3 ledmatrix_mqtt_bridge.py [--config PATH]`, or install +ledmatrix-mqtt-bridge.service. +""" + +from __future__ import annotations + +import argparse +import json +import logging +import os +import signal +import sys +import threading +import time +from typing import Any, Callable, Dict, List, Optional + +import requests + +logger = logging.getLogger("ledmatrix-mqtt-bridge") + +DISCOVERY_PREFIX = "homeassistant" +DEVICE_ID = "ledmatrix" +DEVICE_INFO = { + "identifiers": [DEVICE_ID], + "name": "LEDMatrix", + "manufacturer": "ChuckBuilds", + "model": "LEDMatrix Display", +} + +DEFAULTS = { + "mqtt_host": "localhost", + "mqtt_port": 1883, + "mqtt_username": None, + "mqtt_password": None, + "mqtt_client_id": "ledmatrix-mqtt-bridge", + "mqtt_topic": "ledmatrix/command", + "mqtt_tls": False, + "mqtt_tls_insecure": False, + "ledmatrix_api_base": "http://localhost:5000", + "request_timeout": 15, + "on_demand_duration": None, + "log_level": "INFO", +} + + +class ConfigError(Exception): + """The bridge cannot start with the configuration it was given.""" + + +def load_config(path: str) -> Dict[str, Any]: + """Read bridge_config.json, overlaid on DEFAULTS. + + Every value may also come from the environment as LEDMATRIX_MQTT_, + which is how a password stays out of a file that has to be world-readable + for the service user. + """ + config = dict(DEFAULTS) + if os.path.isfile(path): + with open(path, encoding="utf-8") as handle: + try: + loaded = json.load(handle) + except json.JSONDecodeError as err: + raise ConfigError(f"{path} is not valid JSON: {err}") from err + if not isinstance(loaded, dict): + raise ConfigError(f"{path} must contain a JSON object") + config.update(loaded) + else: + logger.warning("No config file at %s - using defaults and environment", path) + + for key in DEFAULTS: + env_value = os.environ.get(f"LEDMATRIX_MQTT_{key.upper()}") + if env_value is not None: + config[key] = env_value + + for key in ("mqtt_port", "request_timeout"): + try: + config[key] = int(config[key]) + except (TypeError, ValueError) as err: + raise ConfigError(f"{key} must be a whole number, got {config[key]!r}") from err + for key in ("mqtt_tls", "mqtt_tls_insecure"): + config[key] = str(config[key]).lower() in ("1", "true", "yes", "on") + + if config.get("mqtt_password") == "REPLACE_WITH_YOUR_ACTUAL_MQTT_PASSWORD": + raise ConfigError( + "mqtt_password is still the example placeholder - set a real password, " + "or remove the key if your broker allows anonymous connections") + return config + + +class LEDMatrixClient: + """The api_v3 calls the bridge needs, and nothing else. + + Everything goes through the HTTP API rather than the filesystem, so the + bridge does not have to live on the Pi, does not need read access to + config.json, and cannot drift from the web UI's own behaviour. + """ + + def __init__(self, api_base: str, timeout: int = 15, + session: Optional[requests.Session] = None): + self.api_base = api_base.rstrip("/") + self.timeout = timeout + self.session = session or requests.Session() + + def _call(self, method: str, path: str, **kwargs) -> Dict[str, Any]: + url = f"{self.api_base}/api/v3{path}" + response = self.session.request(method, url, timeout=self.timeout, **kwargs) + try: + body = response.json() + except ValueError: + body = {} + if response.status_code >= 400 or body.get("status") == "error": + message = body.get("message") or f"HTTP {response.status_code}" + raise RuntimeError(f"{method} {path} failed: {message}") + return body.get("data", body) + + def list_modes(self) -> List[Dict[str, Any]]: + """Every display mode that can be force-displayed, newest discovery. + + /display/modes triggers plugin discovery itself, which matters because + discovery is lazy: a bridge that never opens the dashboard would + otherwise see nothing at all. + """ + return self._call("GET", "/display/modes").get("modes", []) + + def display_status(self) -> Dict[str, Any]: + return self._call("GET", "/display/on-demand/status") + + def start_on_demand(self, mode: str, plugin_id: Optional[str] = None, + duration: Optional[int] = None, pinned: bool = False) -> Dict[str, Any]: + payload: Dict[str, Any] = {"mode": mode, "pinned": pinned} + if plugin_id: + # find_plugin_for_mode only sees modes declared in a static + # manifest, so a plugin whose modes are generated -- each installed + # Starlark app is one -- 404s when plugin_id is omitted. Sending it + # skips that lookup. /display/modes reports it for every mode. + payload["plugin_id"] = plugin_id + if duration: + payload["duration"] = int(duration) + return self._call("POST", "/display/on-demand/start", json=payload) + + def stop_on_demand(self) -> Dict[str, Any]: + return self._call("POST", "/display/on-demand/stop", json={}) + + def set_power(self, on: bool) -> Dict[str, Any]: + action = "start_display" if on else "stop_display" + return self._call("POST", "/system/action", json={"action": action}) + + def get_brightness(self) -> Optional[int]: + config = self._call("GET", "/config/main") + value = config.get("display", {}).get("hardware", {}).get("brightness") + try: + return int(value) + except (TypeError, ValueError): + return None + + def set_brightness(self, value: int) -> Dict[str, Any]: + return self._call("POST", "/config/main", json={"brightness": int(value)}) + + +class CommandHandler: + """Turns one decoded MQTT payload into one API call. + + Kept free of MQTT so it can be tested against a fake client: the failure + modes worth pinning are all in here (an unknown mode, an out-of-range + brightness, a mode name that needs its plugin_id attached). + """ + + def __init__(self, client: LEDMatrixClient, default_duration: Optional[int] = None): + self.client = client + self.default_duration = default_duration + self._modes_by_name: Dict[str, Dict[str, Any]] = {} + + def refresh_modes(self) -> List[Dict[str, Any]]: + modes = self.client.list_modes() + self._modes_by_name = {m["mode"]: m for m in modes} + # Home Assistant's select shows labels, so accept them back as well -- + # otherwise picking "Simple Clock" in a dashboard is not a mode name. + for entry in modes: + self._modes_by_name.setdefault(entry.get("name") or entry["mode"], entry) + return modes + + @property + def known_modes(self) -> List[Dict[str, Any]]: + return list({id(v): v for v in self._modes_by_name.values()}.values()) + + def handle(self, payload: Dict[str, Any]) -> Dict[str, Any]: + action = payload.get("action") + handlers: Dict[str, Callable[[Dict[str, Any]], Dict[str, Any]]] = { + "display": self._display, + "stop_display": lambda _p: self._ok(self.client.stop_on_demand()), + "power": self._power, + "brightness": self._brightness, + "refresh": lambda _p: self._ok({"modes": len(self.refresh_modes())}), + } + handler = handlers.get(action) + if handler is None: + return self._error(f"Unknown action {action!r}; expected one of " + f"{', '.join(sorted(handlers))}") + try: + return handler(payload) + except (requests.RequestException, RuntimeError) as err: + logger.error("Command %s failed: %s", action, err) + return self._error(str(err)) + + def _display(self, payload: Dict[str, Any]) -> Dict[str, Any]: + mode = payload.get("mode") + plugin_id = payload.get("plugin_id") + if not mode and not plugin_id: + return self._error("display requires 'mode' or 'plugin_id'") + + known = self._modes_by_name.get(mode) if mode else None + if known is None and mode and not plugin_id: + # One retry against a fresh listing: a plugin installed since the + # last refresh is the common reason a valid mode looks unknown. + self.refresh_modes() + known = self._modes_by_name.get(mode) + if known is not None: + mode = known["mode"] + plugin_id = plugin_id or known.get("plugin_id") + + duration = payload.get("duration", self.default_duration) + pinned = bool(payload.get("pinned", False)) + result = self.client.start_on_demand( + mode=mode, plugin_id=plugin_id, duration=duration, pinned=pinned) + return self._ok(result, mode=mode, plugin_id=plugin_id) + + def _power(self, payload: Dict[str, Any]) -> Dict[str, Any]: + state = str(payload.get("state", "")).strip().lower() + if state not in ("on", "off"): + return self._error("power requires 'state' of 'on' or 'off'") + return self._ok(self.client.set_power(state == "on"), state=state) + + def _brightness(self, payload: Dict[str, Any]) -> Dict[str, Any]: + raw = payload.get("value") + try: + value = int(float(raw)) + except (TypeError, ValueError): + return self._error(f"brightness requires a number, got {raw!r}") + if not 0 <= value <= 100: + return self._error(f"brightness must be between 0 and 100, got {value}") + return self._ok(self.client.set_brightness(value), value=value) + + @staticmethod + def _ok(result: Any, **extra) -> Dict[str, Any]: + return {"status": "success", "result": result, **extra} + + @staticmethod + def _error(message: str) -> Dict[str, Any]: + return {"status": "error", "message": message} + + +def discovery_messages(command_topic: str, state_topic: str, availability_topic: str, + mode_labels: List[str]) -> List[Dict[str, Any]]: + """The retained MQTT Discovery configs, as {topic, payload} pairs. + + Pure, so the entity shapes can be asserted without a broker. Every entity + shares one availability topic, which is also the bridge's last will -- HA + then shows the matrix as unavailable when the bridge dies, instead of + leaving stale controls that silently do nothing. + """ + common = { + "device": DEVICE_INFO, + "availability_topic": availability_topic, + "payload_available": "online", + "payload_not_available": "offline", + } + return [ + { + "topic": f"{DISCOVERY_PREFIX}/select/{DEVICE_ID}/display_mode/config", + "payload": { + **common, + "name": "Display Mode", + "unique_id": f"{DEVICE_ID}_display_mode", + "command_topic": command_topic, + "command_template": '{"action": "display", "mode": "{{ value }}"}', + "state_topic": state_topic, + "value_template": "{{ value_json.mode }}", + "options": mode_labels, + "icon": "mdi:view-dashboard", + }, + }, + { + "topic": f"{DISCOVERY_PREFIX}/button/{DEVICE_ID}/stop_display/config", + "payload": { + **common, + "name": "Stop Display", + "unique_id": f"{DEVICE_ID}_stop_display", + "command_topic": command_topic, + "payload_press": '{"action": "stop_display"}', + "icon": "mdi:stop", + }, + }, + { + "topic": f"{DISCOVERY_PREFIX}/switch/{DEVICE_ID}/power/config", + "payload": { + **common, + "name": "Power", + "unique_id": f"{DEVICE_ID}_power", + "command_topic": command_topic, + "payload_on": '{"action": "power", "state": "on"}', + "payload_off": '{"action": "power", "state": "off"}', + "state_topic": state_topic, + "value_template": "{{ 'ON' if value_json.power else 'OFF' }}", + "state_on": "ON", + "state_off": "OFF", + "icon": "mdi:power", + }, + }, + { + "topic": f"{DISCOVERY_PREFIX}/number/{DEVICE_ID}/brightness/config", + "payload": { + **common, + "name": "Brightness", + "unique_id": f"{DEVICE_ID}_brightness", + "command_topic": command_topic, + "command_template": '{"action": "brightness", "value": {{ value }}}', + "state_topic": state_topic, + "value_template": "{{ value_json.brightness }}", + "min": 0, + "max": 100, + "step": 1, + "icon": "mdi:brightness-6", + }, + }, + ] + + +def read_state(client: LEDMatrixClient) -> Dict[str, Any]: + """The state every entity reads, so HA opens on real values. + + Each field is fetched independently: a matrix with its display service + stopped still has a brightness worth showing, and one unreachable field + should not blank the rest. + """ + state: Dict[str, Any] = {"power": False, "mode": None, "brightness": None} + try: + status = client.display_status() + state["power"] = bool(status.get("service", {}).get("active")) + on_demand = status.get("state", {}) + if on_demand.get("active"): + state["mode"] = on_demand.get("mode") + except (requests.RequestException, RuntimeError) as err: + logger.debug("Could not read display status: %s", err) + try: + state["brightness"] = client.get_brightness() + except (requests.RequestException, RuntimeError) as err: + logger.debug("Could not read brightness: %s", err) + return state + + +class Bridge: + """MQTT wiring around CommandHandler.""" + + def __init__(self, config: Dict[str, Any]): + self.config = config + self.command_topic = config["mqtt_topic"] + self.status_topic = f"{self.command_topic}/status" + self.state_topic = f"{self.command_topic}/state" + self.availability_topic = f"{self.command_topic}/availability" + self.client = LEDMatrixClient(config["ledmatrix_api_base"], config["request_timeout"]) + self.handler = CommandHandler(self.client, config.get("on_demand_duration")) + self._stop = threading.Event() + self._mqtt = None + + # -- MQTT callbacks (paho-mqtt 2.x VERSION2 signatures) ------------------ + + def _on_connect(self, client, _userdata, _flags, reason_code, _properties=None): + if getattr(reason_code, "is_failure", reason_code != 0): + logger.error("MQTT connection refused: %s", reason_code) + return + logger.info("Connected to MQTT broker; subscribing to %s", self.command_topic) + client.subscribe(self.command_topic, qos=1) + client.publish(self.availability_topic, "online", qos=1, retain=True) + # Re-publish on every reconnect, not just the first connect: a broker + # restart drops retained discovery configs, and HA would otherwise be + # left with entities it can no longer describe. + self.publish_discovery() + self.publish_state() + + def _on_message(self, _client, _userdata, message): + try: + payload = json.loads(message.payload.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as err: + logger.warning("Ignoring unparseable message on %s: %s", message.topic, err) + self._publish(self.status_topic, {"status": "error", "message": f"bad payload: {err}"}) + return + if not isinstance(payload, dict): + self._publish(self.status_topic, + {"status": "error", "message": "payload must be a JSON object"}) + return + + logger.info("Command: %s", payload) + result = self.handler.handle(payload) + self._publish(self.status_topic, result) + # The API applies changes asynchronously (the controller polls its + # mailbox), so read state back rather than assuming the command took. + self.publish_state() + + # -- publishing --------------------------------------------------------- + + def _publish(self, topic: str, payload: Any, retain: bool = False) -> None: + if self._mqtt is None: + return + body = payload if isinstance(payload, str) else json.dumps(payload) + self._mqtt.publish(topic, body, qos=1, retain=retain) + + def publish_discovery(self) -> None: + try: + modes = self.handler.refresh_modes() + except (requests.RequestException, RuntimeError) as err: + logger.error("Could not list display modes: %s", err) + modes = self.handler.known_modes + labels = sorted({m.get("name") or m["mode"] for m in modes}) + for message in discovery_messages(self.command_topic, self.state_topic, + self.availability_topic, labels): + self._publish(message["topic"], message["payload"], retain=True) + logger.info("Published discovery for %d display mode(s)", len(labels)) + + def publish_state(self) -> None: + self._publish(self.state_topic, read_state(self.client), retain=True) + + # -- lifecycle ---------------------------------------------------------- + + def run(self) -> int: + try: + import paho.mqtt.client as mqtt + except ImportError: + logger.error("paho-mqtt is not installed: pip install -r requirements.txt") + return 1 + + # VERSION2 is the current callback API. The compatibility note in + # CLAUDE.md is about code written against the v1 signatures; this file + # is written against v2 and requires paho-mqtt >= 2.0. + self._mqtt = mqtt.Client( + mqtt.CallbackAPIVersion.VERSION2, + client_id=self.config["mqtt_client_id"]) + if self.config.get("mqtt_username"): + self._mqtt.username_pw_set(self.config["mqtt_username"], + self.config.get("mqtt_password")) + if self.config.get("mqtt_tls"): + self._mqtt.tls_set() + if self.config.get("mqtt_tls_insecure"): + logger.warning("TLS certificate verification is disabled (mqtt_tls_insecure)") + self._mqtt.tls_insecure_set(True) + + self._mqtt.will_set(self.availability_topic, "offline", qos=1, retain=True) + self._mqtt.on_connect = self._on_connect + self._mqtt.on_message = self._on_message + + logger.info("Connecting to %s:%s", self.config["mqtt_host"], self.config["mqtt_port"]) + try: + self._mqtt.connect(self.config["mqtt_host"], self.config["mqtt_port"], keepalive=60) + except OSError as err: + logger.error("Could not reach the MQTT broker: %s", err) + return 1 + + self._mqtt.loop_start() + try: + while not self._stop.wait(30): + # HA is told the truth about state that changed outside the + # bridge -- somebody using the web UI, or an on-demand window + # expiring on its own. + self.publish_state() + finally: + self._publish(self.availability_topic, "offline", retain=True) + self._mqtt.loop_stop() + self._mqtt.disconnect() + return 0 + + def stop(self, *_args) -> None: + logger.info("Shutting down") + self._stop.set() + + +def main(argv: Optional[List[str]] = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument( + "--config", + default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "bridge_config.json"), + help="Path to bridge_config.json (default: alongside this script)") + args = parser.parse_args(argv) + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s") + try: + config = load_config(args.config) + except ConfigError as err: + logger.error("%s", err) + return 1 + logging.getLogger().setLevel(str(config.get("log_level", "INFO")).upper()) + + bridge = Bridge(config) + signal.signal(signal.SIGTERM, bridge.stop) + signal.signal(signal.SIGINT, bridge.stop) + return bridge.run() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/integrations/mqtt_bridge/requirements.txt b/integrations/mqtt_bridge/requirements.txt new file mode 100644 index 000000000..7bd6cac8d --- /dev/null +++ b/integrations/mqtt_bridge/requirements.txt @@ -0,0 +1,2 @@ +paho-mqtt>=2.0.0 +requests>=2.31.0 diff --git a/plugin-repos/starlark-apps/manager.py b/plugin-repos/starlark-apps/manager.py index fad2dc7d4..4f370a85d 100644 --- a/plugin-repos/starlark-apps/manager.py +++ b/plugin-repos/starlark-apps/manager.py @@ -194,6 +194,15 @@ class StarlarkAppsPlugin(BasePlugin): Each installed app becomes a dynamic display mode. """ + #: Starlark apps are animations: a .webp render carries per-frame delays + #: and _display_frame advances at most one frame per call. The controller + #: reads this attribute to decide whether a mode needs its high-FPS loop; + #: without it display() was called once per rotation slot, so a multi-frame + #: app showed a single frame and never moved. static-image is force-run at + #: high FPS for the same reason (GIFs), but that plugin is special-cased by + #: name in the controller and this one has to declare it. + enable_scrolling = True + def __init__(self, plugin_id: str, config: Dict[str, Any], display_manager, cache_manager, plugin_manager): """Initialize the Starlark Apps plugin.""" @@ -681,13 +690,20 @@ def update(self) -> None: if app.is_enabled() and app.should_render(current_time): self._render_app(app, force=False) - def display(self, force_clear: bool = False) -> bool: + def display(self, display_mode: Optional[str] = None, force_clear: bool = False) -> bool: """ Display current Starlark app. This method is called during the display rotation. Displays frames from the currently active app. + `display_mode` names the app to show when it matches an installed + app_id. The controller passes the mode it is rotating to and inspects + this signature to decide whether to, so accepting it is what lets a + specific app be addressed -- including by an on-demand request pinned + to one app. Anything else (the plugin id itself, when the plugin + exposes no per-app modes) falls through to normal rotation. + Returns False when there is no app to show -- which is the state of every install without Pixlet, and of a fresh one before any app is added. The display controller only skips a mode on a boolean False @@ -698,8 +714,15 @@ def display(self, force_clear: bool = False) -> bool: if force_clear: self.display_manager.clear() - # If no current app, try to select one - if not self.current_app: + if display_mode and display_mode in self.apps: + self.current_app = self.apps[display_mode] + elif force_clear or not self.current_app: + # Advance on entry to the mode. _select_next_app only ran when + # current_app was unset, so the first enabled app was picked + # once and then shown forever -- every other installed app was + # rendered on schedule and never displayed. force_clear is the + # controller's "we just switched to you" signal (it is reset + # immediately after this call), so one app gets each turn. self._select_next_app() if not self.current_app: diff --git a/plugin-repos/starlark-apps/pixlet_renderer.py b/plugin-repos/starlark-apps/pixlet_renderer.py index 40f8f59d0..29d75611c 100644 --- a/plugin-repos/starlark-apps/pixlet_renderer.py +++ b/plugin-repos/starlark-apps/pixlet_renderer.py @@ -264,10 +264,18 @@ def render( else: value_str = str(value) - # Validate value doesn't contain dangerous shell metacharacters - # Block: backticks, $(), pipes, redirects, semicolons, ampersands, null bytes - # Allow: most printable chars including spaces, quotes, brackets, braces - if re.search(r'[`$|<>&;\x00]|\$\(', value_str): + # Validate value doesn't contain dangerous shell metacharacters. + # Kept as defence in depth only: cmd is a list and there is no + # shell=True below, so nothing here is ever interpreted by a + # shell. That made the list worth trimming rather than growing + # -- "|" is a normal character inside a config value, and apps + # do use it as a separator (a PennDOT sign id is + # "I-476 North|175659"). Blocking it dropped the whole key + # silently, and the app then rendered its own "not configured" + # screen with nothing to say why. + # Block: backticks, $(), redirects, semicolons, ampersands, null bytes + # Allow: most printable chars including spaces, quotes, brackets, braces, pipes + if re.search(r'[`$<>&;\x00]|\$\(', value_str): logger.warning(f"Skipping config value with unsafe shell characters for key {key}: {value_str}") continue @@ -299,13 +307,21 @@ def render( ) if result.returncode == 0: - if os.path.isfile(output_path): - logger.debug(f"Successfully rendered: {star_file} -> {output_path}") - return True, None - else: + if not os.path.isfile(output_path): error = "Rendering succeeded but output file not found" logger.error(error) return False, error + # Pixlet exits 0 and writes a 0-byte file when the app renders + # nothing -- an app whose config leaves it with no content to + # show does exactly that. Treating existence alone as success + # handed the caller a file with no frames in it, which reads + # downstream as a working app that draws a black panel. + if os.path.getsize(output_path) == 0: + error = "Rendering produced an empty (0-byte) file - the app rendered no content" + logger.error(error) + return False, error + logger.debug(f"Successfully rendered: {star_file} -> {output_path}") + return True, None else: error = f"Pixlet failed (exit {result.returncode}): {result.stderr}" logger.error(error) @@ -319,11 +335,76 @@ def render( logger.exception("Rendering exception") return False, "Rendering failed - see logs for details" + #: Schema extraction runs an app's own get_schema(), which may make a + #: network call. Short enough that a hung app does not stall an upload, + #: long enough for a real API round trip on a slow connection. + SCHEMA_TIMEOUT = 20 + + def extract_schema_via_pixlet(self, star_file: str) -> Optional[Dict[str, Any]]: + """Ask Pixlet itself for the app's schema, or None if it cannot say. + + `pixlet schema` executes get_schema() instead of reading it, which is + the only way to see options an app computes at runtime -- a dropdown + whose choices come from a live API call has no option list anywhere in + the source for the regex parser below to find, so that parser reports + an empty dropdown and the config form offers nothing to pick. + + Pixlet's own field keys are remapped to the ones the rest of this + plugin and the config UI already use ("typeOf"/"desc"), so the two + extractors return the same shape and callers cannot tell them apart. + """ + if not self.pixlet_binary: + return None + try: + result = subprocess.run( + [self.pixlet_binary, "schema", star_file], + capture_output=True, text=True, timeout=self.SCHEMA_TIMEOUT, + cwd=self._get_safe_working_directory(star_file), + ) + except subprocess.TimeoutExpired: + logger.warning( + "pixlet schema timed out after %ss for %s - get_schema() may be " + "making a slow network call", self.SCHEMA_TIMEOUT, star_file) + return None + except (subprocess.SubprocessError, OSError) as e: + logger.warning(f"Could not run pixlet schema for {star_file}: {e}") + return None + + if result.returncode != 0: + # Not an error worth failing on: older Pixlet builds have no + # `schema` subcommand at all, and the source parser still works. + logger.debug( + "pixlet schema exited %d for %s: %s", + result.returncode, star_file, (result.stderr or '').strip()[:300]) + return None + + try: + schema = json.loads(result.stdout) + except (json.JSONDecodeError, ValueError) as e: + logger.warning(f"pixlet schema returned unparseable output for {star_file}: {e}") + return None + + if not isinstance(schema, dict) or not isinstance(schema.get("schema"), list): + logger.warning(f"pixlet schema returned an unexpected shape for {star_file}") + return None + + for field in schema["schema"]: + if not isinstance(field, dict): + continue + if "type" in field and "typeOf" not in field: + field["typeOf"] = field.pop("type") + if "description" in field and "desc" not in field: + field["desc"] = field.pop("description") + return schema + def extract_schema(self, star_file: str) -> Tuple[bool, Optional[Dict[str, Any]], Optional[str]]: """ - Extract configuration schema from a .star file by parsing source code. + Extract configuration schema from a .star file. - Supports: + Prefers `pixlet schema`, which runs the app and therefore sees options + it computes at runtime. Falls back to parsing the source when Pixlet is + unavailable, too old to have the subcommand, or the app fails to run -- + that parser handles: - Static field definitions (location, text, toggle, dropdown, color, datetime) - Variable-referenced dropdown options - Graceful degradation for unsupported field types @@ -337,6 +418,13 @@ def extract_schema(self, star_file: str) -> Tuple[bool, Optional[Dict[str, Any]] if not os.path.isfile(star_file): return False, None, f"Star file not found: {star_file}" + schema = self.extract_schema_via_pixlet(star_file) + if schema is not None: + logger.debug( + "Extracted schema with %d field(s) from %s via pixlet schema", + len(schema.get('schema', [])), star_file) + return True, schema, None + try: # Read .star file with open(star_file, 'r', encoding='utf-8') as f: diff --git a/scripts/install/install_dns_fix.sh b/scripts/install/install_dns_fix.sh new file mode 100644 index 000000000..94975f0de --- /dev/null +++ b/scripts/install/install_dns_fix.sh @@ -0,0 +1,59 @@ +#!/bin/bash + +# DNS single-request fix installation script. +# +# Optional. Install this only if plugins that call external APIs (Starlark +# apps, weather, sports, music) are timing out or feel slow to first paint +# while the network is otherwise fine. See the header of +# scripts/utils/apply_dns_single_request.sh for what it changes and why. + +set -e + +PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) +SERVICE_NAME="ledmatrix-dns-fix" +UNIT_SRC="$PROJECT_ROOT_DIR/systemd/$SERVICE_NAME.service" +UNIT_DEST="/etc/systemd/system/$SERVICE_NAME.service" + +if [ "$EUID" -eq 0 ]; then + SYSTEMCTL_CMD="systemctl" + SUDO="" +else + SYSTEMCTL_CMD="sudo systemctl" + SUDO="sudo" +fi + +echo "Installing LED Matrix DNS fix service" +echo "Project root directory: $PROJECT_ROOT_DIR" + +if [ ! -f "$UNIT_SRC" ]; then + echo "✗ Missing unit file: $UNIT_SRC" + exit 1 +fi + +chmod +x "$PROJECT_ROOT_DIR/scripts/utils/apply_dns_single_request.sh" + +echo "Installing $UNIT_DEST..." +sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ + | $SUDO tee "$UNIT_DEST" > /dev/null + +$SYSTEMCTL_CMD daemon-reload +$SYSTEMCTL_CMD enable "$SERVICE_NAME.service" +$SYSTEMCTL_CMD start "$SERVICE_NAME.service" || echo "⚠ Failed to start service (will retry on reboot)" + +echo "" +if grep -qs "^options single-request$" /etc/resolv.conf; then + echo "✓ 'options single-request' is active in /etc/resolv.conf" +else + echo "⚠ 'options single-request' is not in /etc/resolv.conf yet." + echo " Check what the service reported:" + echo " journalctl -u $SERVICE_NAME -n 20" +fi + +echo "" +echo "DNS fix installation complete." +echo "" +echo "Useful commands:" +echo " sudo systemctl status $SERVICE_NAME # Check status" +echo " sudo journalctl -u $SERVICE_NAME -n 50 # View logs" +echo " sudo systemctl disable --now $SERVICE_NAME # Undo (edit /etc/resolv.conf to remove the line)" +echo "" diff --git a/scripts/install/install_mqtt_bridge.sh b/scripts/install/install_mqtt_bridge.sh new file mode 100644 index 000000000..7e65ed08f --- /dev/null +++ b/scripts/install/install_mqtt_bridge.sh @@ -0,0 +1,64 @@ +#!/bin/bash + +# Home Assistant MQTT bridge installation script. +# +# Optional. Installs integrations/mqtt_bridge as a service so Home Assistant +# can force display modes, toggle power and set brightness over MQTT. +# See integrations/mqtt_bridge/README.md. + +set -e + +PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) +BRIDGE_DIR="$PROJECT_ROOT_DIR/integrations/mqtt_bridge" +SERVICE_NAME="ledmatrix-mqtt-bridge" +UNIT_SRC="$PROJECT_ROOT_DIR/systemd/$SERVICE_NAME.service" +UNIT_DEST="/etc/systemd/system/$SERVICE_NAME.service" + +if [ "$EUID" -eq 0 ]; then + SYSTEMCTL_CMD="systemctl" + SUDO="" +else + SYSTEMCTL_CMD="sudo systemctl" + SUDO="sudo" +fi + +echo "Installing LED Matrix MQTT bridge" +echo "Project root directory: $PROJECT_ROOT_DIR" + +if [ ! -f "$BRIDGE_DIR/bridge_config.json" ]; then + cp "$BRIDGE_DIR/bridge_config.example.json" "$BRIDGE_DIR/bridge_config.json" + chmod 600 "$BRIDGE_DIR/bridge_config.json" + echo "" + echo "⚠ Created $BRIDGE_DIR/bridge_config.json from the example." + echo " Edit it with your broker details, then re-run this script." + echo " The service will refuse to start until the placeholder password is replaced." + echo "" +fi + +echo "Installing Python dependencies..." +python3 -m pip install -r "$BRIDGE_DIR/requirements.txt" 2>/dev/null \ + || python3 -m pip install --break-system-packages -r "$BRIDGE_DIR/requirements.txt" + +echo "Installing $UNIT_DEST..." +sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ + | $SUDO tee "$UNIT_DEST" > /dev/null + +$SYSTEMCTL_CMD daemon-reload +$SYSTEMCTL_CMD enable "$SERVICE_NAME.service" +$SYSTEMCTL_CMD restart "$SERVICE_NAME.service" || true + +echo "" +if $SYSTEMCTL_CMD is-active --quiet "$SERVICE_NAME.service" 2>/dev/null; then + echo "✓ MQTT bridge is running" + echo " The matrix should appear in Home Assistant under Settings > Devices > MQTT." +else + echo "⚠ MQTT bridge is not running. Check the logs:" + echo " sudo journalctl -u $SERVICE_NAME -n 50" +fi + +echo "" +echo "Useful commands:" +echo " sudo systemctl status $SERVICE_NAME" +echo " sudo journalctl -u $SERVICE_NAME -f" +echo " sudo systemctl disable --now $SERVICE_NAME # Undo" +echo "" diff --git a/scripts/utils/README.md b/scripts/utils/README.md index 53a73f98b..5c9d6843f 100644 --- a/scripts/utils/README.md +++ b/scripts/utils/README.md @@ -9,6 +9,8 @@ This directory contains utility scripts for maintenance and system operations. - **`wifi_monitor_daemon.py`** - Background daemon that monitors WiFi/Ethernet connection and manages access point mode - **`cleanup_venv.sh`** - Cleans up Python virtual environment files - **`clear_python_cache.sh`** - Clears Python cache files (__pycache__, *.pyc, etc.) +- **`pixlet_config_editor.sh`** - Opens Pixlet's own config UI for one installed Starlark app +- **`apply_dns_single_request.sh`** - Adds `options single-request` to the resolver (run by `ledmatrix-dns-fix.service`) ## Usage @@ -25,3 +27,29 @@ This script is typically called by the systemd service (`ledmatrix-web.service`) ### WiFi Monitor Daemon This daemon is typically run as a systemd service (`ledmatrix-wifi-monitor.service`) and automatically manages WiFi access point mode based on network connectivity. + +### Pixlet Config Editor +Run it when you want Pixlet's own config form for a Starlark app -- live +render preview, cascading dropdowns -- rather than the LEDMatrix one. + +```bash +./scripts/utils/pixlet_config_editor.sh # list installed apps +./scripts/utils/pixlet_config_editor.sh penndot_signs # edit, on localhost:8080 +./scripts/utils/pixlet_config_editor.sh penndot_signs --lan # reachable from the LAN +``` + +Deliberately not a service. It stops the display for the length of the +session and `pixlet serve` listens with no authentication, so it should only +be running while you are actually editing. It backs the config up first and +restarts the display on exit, however it exits. + +From another machine, forward the port rather than using `--lan`: + +```bash +ssh -L 8080:localhost:8080 pi@ledpi.local +``` + +### Apply DNS Single-Request Fix +Installed and run by `ledmatrix-dns-fix.service`; see `systemd/README.md`. +Safe to run by hand (`sudo ./scripts/utils/apply_dns_single_request.sh`) and +idempotent. diff --git a/scripts/utils/apply_dns_single_request.sh b/scripts/utils/apply_dns_single_request.sh new file mode 100644 index 000000000..c8dbb1c1d --- /dev/null +++ b/scripts/utils/apply_dns_single_request.sh @@ -0,0 +1,69 @@ +#!/bin/bash +# +# Add `options single-request` to the system resolver configuration. +# +# glibc's getaddrinfo() sends the A and AAAA queries for a name in +# parallel on one socket. Some routers answer the A query and drop the +# AAAA one, so the resolver waits out its full timeout -- about five +# seconds -- before returning an address that was already available. +# Disabling IPv6 in the kernel does not help: the resolver still asks. +# +# `single-request` makes it send the two queries one after the other, +# which those routers answer correctly. Anything on the matrix that +# calls an external API pays that five seconds per lookup otherwise, and +# a Starlark app with a render timeout will simply fail instead. +# +# Idempotent, and safe to run on a machine that does not need it. Run by +# ledmatrix-dns-fix.service on every boot, because whatever manages +# resolv.conf regenerates it and drops the option again. +# +# Usage: sudo ./scripts/utils/apply_dns_single_request.sh + +set -eu + +OPTION="options single-request" +RESOLVCONF_TAIL="/etc/resolvconf/resolv.conf.d/tail" +RESOLV_CONF="/etc/resolv.conf" + +log() { echo "[dns-single-request] $*"; } + +already_applied() { + grep -qs "^${OPTION}\$" "$1" +} + +# resolvconf regenerates /etc/resolv.conf from these fragments, so the +# tail file is the only place an addition survives. Prefer it when the +# directory exists, whether or not resolvconf has run yet. +if [ -d "$(dirname "$RESOLVCONF_TAIL")" ]; then + if already_applied "$RESOLVCONF_TAIL"; then + log "already present in $RESOLVCONF_TAIL" + else + echo "$OPTION" >> "$RESOLVCONF_TAIL" + log "added to $RESOLVCONF_TAIL" + fi + command -v resolvconf >/dev/null 2>&1 && resolvconf -u || true +fi + +# systemd-resolved owns its stub file and rewrites anything appended to +# it. The equivalent setting there is a resolved.conf drop-in, which is +# a different change than this script makes -- say so rather than +# writing to a file that will be overwritten. +if [ -L "$RESOLV_CONF" ] && readlink -f "$RESOLV_CONF" | grep -q "systemd"; then + log "$RESOLV_CONF is managed by systemd-resolved; not modifying it." + log "If lookups are slow, see 'man resolved.conf' -- the resolver options" + log "there are set per-link, not through resolv.conf." + exit 0 +fi + +if already_applied "$RESOLV_CONF"; then + log "already present in $RESOLV_CONF" + exit 0 +fi + +if [ -w "$RESOLV_CONF" ] || [ ! -e "$RESOLV_CONF" ]; then + echo "$OPTION" >> "$RESOLV_CONF" + log "added to $RESOLV_CONF" +else + log "cannot write $RESOLV_CONF (run with sudo?)" + exit 1 +fi diff --git a/scripts/utils/pixlet_config_editor.sh b/scripts/utils/pixlet_config_editor.sh new file mode 100644 index 000000000..7c57de47b --- /dev/null +++ b/scripts/utils/pixlet_config_editor.sh @@ -0,0 +1,148 @@ +#!/bin/bash +# +# Edit an installed Starlark app's config in Pixlet's own config UI. +# +# `pixlet serve` runs the app for real, so its form has working cascading +# dropdowns and option lists fetched live -- useful for an app whose choices +# only exist at runtime, or when you want to see the render change as you +# type. The LEDMatrix config form now reads the same runtime schema (see +# PixletRenderer.extract_schema_via_pixlet), so reach for this when you want +# Pixlet's live preview, not because the normal form is missing options. +# +# Deliberately a script you run and then Ctrl+C, not a service: it stops the +# display for the length of the session, and `pixlet serve` listens on a port +# with no authentication. Nothing here should be listening when you are not +# actually editing. +# +# Usage: +# ./scripts/utils/pixlet_config_editor.sh # list installed apps +# ./scripts/utils/pixlet_config_editor.sh # edit, on localhost +# ./scripts/utils/pixlet_config_editor.sh --lan # reachable from the LAN +# +# On localhost, reach it from another machine over SSH instead of --lan: +# ssh -L 8080:localhost:8080 pi@ledpi.local + +set -eu + +PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) +APPS_DIR="$PROJECT_ROOT_DIR/starlark-apps" +PORT="${PIXLET_EDITOR_PORT:-8080}" +BIND_HOST="127.0.0.1" + +APP_ID="${1:-}" +if [ "${2:-}" = "--lan" ]; then + BIND_HOST="0.0.0.0" +fi + +list_apps() { + if [ -d "$APPS_DIR" ]; then + find "$APPS_DIR" -maxdepth 1 -mindepth 1 -type d -printf ' %f\n' 2>/dev/null | sort + fi +} + +if [ -z "$APP_ID" ]; then + echo "Usage: $0 [--lan]" + echo "" + echo "Installed apps:" + list_apps || true + [ -n "$(list_apps)" ] || echo " (none found in $APPS_DIR)" + exit 1 +fi + +APP_DIR="$APPS_DIR/$APP_ID" +if [ ! -d "$APP_DIR" ]; then + echo "No such app: $APP_ID" + echo "" + echo "Installed apps:" + list_apps + exit 1 +fi + +STAR_FILE=$(find "$APP_DIR" -maxdepth 1 -iname "*.star" | head -1) +if [ -z "$STAR_FILE" ]; then + echo "No .star file found in $APP_DIR" + exit 1 +fi + +# Same search order the plugin itself uses: the bundled binary for this +# architecture first, then PATH -- so this works on an install that never put +# pixlet on PATH. +find_pixlet() { + local arch bundled + case "$(uname -s)-$(uname -m)" in + Linux-aarch64|Linux-arm64) arch="pixlet-linux-arm64" ;; + Linux-x86_64|Linux-amd64) arch="pixlet-linux-amd64" ;; + Darwin-arm64) arch="pixlet-darwin-arm64" ;; + Darwin-x86_64) arch="pixlet-darwin-amd64" ;; + *) arch="" ;; + esac + bundled="$PROJECT_ROOT_DIR/bin/pixlet/$arch" + if [ -n "$arch" ] && [ -x "$bundled" ]; then + echo "$bundled" + return 0 + fi + command -v pixlet 2>/dev/null || return 1 +} + +PIXLET_BIN=$(find_pixlet) || { + echo "Pixlet not found. Install it with:" + echo " ./scripts/download_pixlet.sh" + exit 1 +} + +CONFIG_FILE="$APP_DIR/config.json" +if [ -f "$CONFIG_FILE" ]; then + cp "$CONFIG_FILE" "$CONFIG_FILE.backup" + echo "Backed up existing config to $CONFIG_FILE.backup" +else + echo "{}" > "$CONFIG_FILE" +fi + +DISPLAY_WAS_RUNNING=false +if systemctl is-active --quiet ledmatrix 2>/dev/null; then + DISPLAY_WAS_RUNNING=true +fi + +# Restart the display however this exits -- Ctrl+C, an error, or pixlet +# dying on its own. Leaving the panel dark because the editor crashed is the +# failure worth guarding against. +cleanup() { + echo "" + if [ "$DISPLAY_WAS_RUNNING" = true ]; then + echo "Restarting the display service..." + sudo systemctl restart ledmatrix || echo "⚠ Could not restart ledmatrix - do it by hand" + fi + echo "Your config as it was before this session: $CONFIG_FILE.backup" +} +trap cleanup EXIT INT TERM + +if [ "$DISPLAY_WAS_RUNNING" = true ]; then + echo "Stopping the display service so it does not read config.json mid-write..." + sudo systemctl stop ledmatrix +fi + +echo "" +echo "Editing: $APP_ID" +echo "App file: $STAR_FILE" +if [ "$BIND_HOST" = "0.0.0.0" ]; then + echo "URL: http://$(hostname):$PORT/" + echo "" + echo "⚠ Listening on all interfaces with no authentication. Anyone on this" + echo " network can change this app's config while the session is open." +else + echo "URL: http://localhost:$PORT/" + echo "" + echo "Listening on localhost only. From another machine, forward the port:" + echo " ssh -L $PORT:localhost:$PORT $(whoami)@$(hostname)" +fi +echo "" +echo "Changes save straight to the real config as you make them." +echo "Press Ctrl+C when finished - the display restarts automatically." +echo "" + +cd "$APP_DIR" +"$PIXLET_BIN" serve "$(basename "$STAR_FILE")" \ + --host "$BIND_HOST" \ + --port "$PORT" \ + --no-browser \ + --saveconfig "$CONFIG_FILE" diff --git a/src/display_controller.py b/src/display_controller.py index 04dbe79d2..8a9025707 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -307,40 +307,8 @@ def _follower_gated_update(): # Check for on-demand plugin filter from cache on_demand_config = self.cache_manager.get('display_on_demand_config', max_age=3600) - on_demand_plugin_id = on_demand_config.get('plugin_id') if on_demand_config else None + enabled_plugins = self._select_startup_plugins(discovered_plugins, on_demand_config) - if on_demand_plugin_id: - logger.info("On-demand mode detected during initialization: filtering to plugin '%s' only", on_demand_plugin_id) - # Only load the on-demand plugin, but ensure it's enabled - if on_demand_plugin_id not in discovered_plugins: - error_msg = f"On-demand plugin '{on_demand_plugin_id}' not found in discovered plugins" - logger.error(error_msg) - logger.warning("Falling back to normal mode (all enabled plugins)") - on_demand_plugin_id = None - enabled_plugins = [p for p in discovered_plugins if self.config.get(p, {}).get('enabled', False)] - else: - plugin_config = self.config.get(on_demand_plugin_id, {}) - was_disabled = not plugin_config.get('enabled', False) - if was_disabled: - logger.info("Temporarily enabling plugin '%s' for on-demand mode", on_demand_plugin_id) - if on_demand_plugin_id not in self.config: - self.config[on_demand_plugin_id] = {} - self.config[on_demand_plugin_id]['enabled'] = True - enabled_plugins = [on_demand_plugin_id] - # Set on-demand state from cached config - self.on_demand_active = True - self.on_demand_plugin_id = on_demand_plugin_id - self.on_demand_mode = on_demand_config.get('mode') - self.on_demand_duration = on_demand_config.get('duration') - self.on_demand_pinned = on_demand_config.get('pinned', False) - self.on_demand_requested_at = on_demand_config.get('requested_at') - self.on_demand_expires_at = on_demand_config.get('expires_at') - self.on_demand_status = 'active' - self.on_demand_schedule_override = True - logger.info("On-demand mode: loading only plugin '%s'", on_demand_plugin_id) - else: - enabled_plugins = [p for p in discovered_plugins if self.config.get(p, {}).get('enabled', False)] - # Count enabled plugins for progress tracking enabled_count = len(enabled_plugins) logger.info("Loading %d enabled plugin(s) in parallel (max 4 concurrent)...", enabled_count) @@ -1279,6 +1247,91 @@ def _set_on_demand_error(self, message: str) -> None: #: perceptible, and it cuts the read rate by 30x. ON_DEMAND_POLL_INTERVAL = 0.25 + def _select_startup_plugins(self, discovered_plugins: List[str], + on_demand_config: Optional[Dict[str, Any]]) -> List[str]: + """Which plugins to load at startup, restoring on-demand state if any. + + Every normally-enabled plugin loads, on-demand or not. Loading only the + on-demand plugin left every other plugin unavailable for the rest of + the process's life whenever the service was restarted while on-demand + was still active -- and a restart during an on-demand session is + routine, since that is how updates and config changes are applied. The + panel came back cycling that one plugin's modes and nothing else, with + no way out but clearing the on-demand cache by hand. + + On-demand still resumes on its saved mode; this only widens what gets + loaded, so normal rotation has somewhere to return to when it ends. + A plugin that is disabled in config but named by the on-demand request + is still enabled and added, since otherwise the mode being resumed + would have nothing behind it. + """ + enabled_plugins = [p for p in discovered_plugins + if self.config.get(p, {}).get('enabled', False)] + + on_demand_plugin_id = on_demand_config.get('plugin_id') if on_demand_config else None + if not on_demand_plugin_id: + return enabled_plugins + + if on_demand_plugin_id not in discovered_plugins: + logger.error("On-demand plugin '%s' not found in discovered plugins", + on_demand_plugin_id) + logger.warning("Falling back to normal mode (all enabled plugins)") + return enabled_plugins + + if not self.config.get(on_demand_plugin_id, {}).get('enabled', False): + logger.info("Temporarily enabling plugin '%s' for on-demand mode", on_demand_plugin_id) + self.config.setdefault(on_demand_plugin_id, {})['enabled'] = True + if on_demand_plugin_id not in enabled_plugins: + enabled_plugins.append(on_demand_plugin_id) + + # Restore on-demand state from the cached request so it resumes. + self.on_demand_active = True + self.on_demand_plugin_id = on_demand_plugin_id + self.on_demand_mode = on_demand_config.get('mode') + self.on_demand_duration = on_demand_config.get('duration') + self.on_demand_pinned = on_demand_config.get('pinned', False) + self.on_demand_requested_at = on_demand_config.get('requested_at') + self.on_demand_expires_at = on_demand_config.get('expires_at') + self.on_demand_status = 'active' + self.on_demand_schedule_override = True + logger.info("On-demand mode detected during initialization: resuming on plugin '%s'; " + "all %d enabled plugin(s) still load normally", + on_demand_plugin_id, len(enabled_plugins)) + return enabled_plugins + + def _consume_on_demand_request(self, request_id: str) -> None: + """Remove the request we just handled from the mailbox. + + Leaving it on disk meant a restart replayed the previous request: the + fresh controller read it, activated it and cached it, so the request + the caller had just made was ignored and the panel silently showed the + earlier plugin. + + Compare before deleting. The web process can post a newer request + between the read and this delete; an unconditional delete threw that + one away and it was never processed -- the user's second click did + nothing. Re-reading uncached and only deleting our own request_id + leaves a newer request in the mailbox for the next poll instead. + + This narrows the window rather than closing it: a request landing + between the re-read and the delete is still lost. Closing it properly + needs an atomic claim (a rename, or a compare-and-delete primitive) + that the cache layer does not currently offer, so the honest fix is a + smaller window plus this note, not a bigger lock. For start requests + processed_id still guards against reprocessing if the delete fails. + """ + try: + current = self.cache_manager.get('display_on_demand_request', + max_age=3600, memory_ttl=0) + if not current or current.get('request_id') == request_id: + self.cache_manager.delete('display_on_demand_request') + else: + logger.debug("Newer on-demand request %s arrived while processing " + "%s; leaving it in the mailbox", + current.get('request_id'), request_id) + except (OSError, AttributeError, KeyError) as err: + logger.debug("Could not clear the on-demand request mailbox: %s", err) + def _poll_on_demand_requests(self) -> None: """Poll cache for new on-demand requests from external controllers.""" now = time.monotonic() @@ -1325,8 +1378,15 @@ def _poll_on_demand_requests(self) -> None: logger.debug("Stop request %s received but on-demand is not active", request_id) # Still update request_id to acknowledge the request self.on_demand_request_id = request_id + # Stop requests are deliberately exempt from the request_id/ + # processed_id guards above, so that a second click stops a mode + # that a race left running. Consuming the mailbox is therefore the + # only thing that ends the request: without it the same stop was + # re-read and re-processed on every poll, forever, logging at + # ON_DEMAND_POLL_INTERVAL for the life of the process. + self._consume_on_demand_request(request_id) return - + # For start requests, check if already processed if request_id == self.on_demand_request_id: logger.debug("On-demand start request %s already processed (instance check)", request_id) @@ -1344,37 +1404,8 @@ def _poll_on_demand_requests(self) -> None: # Mark as processed BEFORE processing (to prevent duplicate processing) self.cache_manager.set('display_on_demand_processed_id', request_id, ttl=3600) self.on_demand_request_id = request_id - # Consume the mailbox entry. Leaving it on disk meant a restart replayed - # the previous request: the fresh controller read it, activated it and - # cached it, so the request the caller had just made was ignored and the - # panel silently showed the earlier plugin. processed_id still guards - # against double-processing if this delete fails. - try: - # Compare before deleting. The web process can post a newer request - # between the read above and this delete; an unconditional delete - # threw that one away and it was never processed -- the user's - # second click did nothing. Re-reading uncached and only deleting - # our own request_id means a newer request is left in the mailbox - # for the next poll instead. - # - # This narrows the window rather than closing it: a request landing - # between this re-read and the delete is still lost. Closing it - # properly needs an atomic claim (a rename, or a compare-and-delete - # primitive) that the cache layer does not currently offer, so the - # honest fix is a smaller window plus this note, not a bigger lock. - current = self.cache_manager.get('display_on_demand_request', - max_age=3600, memory_ttl=0) - if not current or current.get('request_id') == request_id: - self.cache_manager.delete('display_on_demand_request') - else: - logger.debug("Newer on-demand request %s arrived while processing " - "%s; leaving it in the mailbox", - current.get('request_id'), request_id) - except (OSError, AttributeError, KeyError) as err: - # Best-effort: processed_id still guards against reprocessing if the - # mailbox cannot be cleared. - logger.debug("Could not clear the on-demand request mailbox: %s", err) - + self._consume_on_demand_request(request_id) + if action == 'start': logger.info("Processing on-demand start request for plugin: %s", request.get('plugin_id')) self._activate_on_demand(request) @@ -1416,34 +1447,28 @@ def _resolve_mode_for_plugin(self, plugin_id: Optional[str], mode: Optional[str] return modes[0] return plugin_id - def _populate_on_demand_modes_from_plugin(self) -> None: - """ - Populate on_demand_modes from the on-demand plugin's display modes. - Called after plugin loading completes when on-demand state is restored from cache. + def _on_demand_modes_for_plugin(self, plugin_id: str) -> List[str]: + """Every loaded display mode belonging to `plugin_id`, in rotation order. + + Live modes that actually have content lead, then the rest, then live + modes with nothing to show -- so an on-demand request for a sports + plugin opens on a game in progress rather than an empty live screen. + Returns an empty list when the plugin has no loaded modes. """ - if not self.on_demand_active or not self.on_demand_plugin_id: - return - - plugin_id = self.on_demand_plugin_id - - # Get all modes for this plugin plugin_modes = self.plugin_display_modes.get(plugin_id, []) if not plugin_modes: # Fallback: find all modes that belong to this plugin plugin_modes = [mode for mode, pid in self.mode_to_plugin_id.items() if pid == plugin_id] - + # Filter to only include modes that exist in plugin_modes available_plugin_modes = [m for m in plugin_modes if m in self.plugin_modes] - if not available_plugin_modes: - logger.warning("No valid display modes found for on-demand plugin '%s' after restoration", plugin_id) - self.on_demand_modes = [] - return - + return [] + # Prioritize live modes if they exist and have content live_modes = [m for m in available_plugin_modes if m.endswith('_live')] other_modes = [m for m in available_plugin_modes if not m.endswith('_live')] - + # Check if live modes have content live_with_content = [] for live_mode in live_modes: @@ -1454,18 +1479,57 @@ def _populate_on_demand_modes_from_plugin(self) -> None: live_with_content.append(live_mode) except Exception: pass - + # Build mode list: live modes with content first, then other modes, then live modes without content if live_with_content: ordered_modes = live_with_content + other_modes + [m for m in live_modes if m not in live_with_content] else: # No live content, skip live modes ordered_modes = other_modes - + if not ordered_modes: # Only live modes available but no content - use them anyway ordered_modes = live_modes - + + return ordered_modes + + def _apply_on_demand_pin(self, ordered_modes: List[str], resolved_mode: Optional[str], + pinned: bool) -> List[str]: + """Narrow an on-demand rotation to the single requested mode when pinned. + + `pinned` reaches the controller from the API and was stored and + republished but never acted on, so a pinned request still rotated + through every mode the resolved plugin owns. That is the right default + for a sports plugin, whose modes are views of one subject + (nfl_live/nfl_recent/nfl_upcoming), and the wrong one for a plugin + whose modes are unrelated -- each Starlark app is its own widget, so + asking for one and getting all of them is not what was requested. + """ + if not pinned or not resolved_mode or resolved_mode not in ordered_modes: + return ordered_modes + return [resolved_mode] + + def _populate_on_demand_modes_from_plugin(self) -> None: + """ + Populate on_demand_modes from the on-demand plugin's display modes. + Called after plugin loading completes when on-demand state is restored from cache. + """ + if not self.on_demand_active or not self.on_demand_plugin_id: + return + + plugin_id = self.on_demand_plugin_id + + ordered_modes = self._on_demand_modes_for_plugin(plugin_id) + if not ordered_modes: + logger.warning("No valid display modes found for on-demand plugin '%s' after restoration", plugin_id) + self.on_demand_modes = [] + return + + # A restart must not silently un-pin: the pin is part of the request + # being resumed, and it is restored from the same cached config above. + ordered_modes = self._apply_on_demand_pin( + ordered_modes, self.on_demand_mode, self.on_demand_pinned) + self.on_demand_modes = ordered_modes # Set index to match the restored mode if available, otherwise start at 0 if self.on_demand_mode and self.on_demand_mode in ordered_modes: @@ -1520,46 +1584,14 @@ def _activate_on_demand(self, request: Dict[str, Any]) -> None: if resolved_mode in self.available_modes: self.current_mode_index = self.available_modes.index(resolved_mode) - # Get all modes for this plugin - plugin_modes = self.plugin_display_modes.get(resolved_plugin_id, []) - if not plugin_modes: - # Fallback: find all modes that belong to this plugin - plugin_modes = [mode for mode, pid in self.mode_to_plugin_id.items() if pid == resolved_plugin_id] - - # Filter to only include modes that exist in plugin_modes - available_plugin_modes = [m for m in plugin_modes if m in self.plugin_modes] - - if not available_plugin_modes: + ordered_modes = self._on_demand_modes_for_plugin(resolved_plugin_id) + if not ordered_modes: logger.error("No valid display modes found for plugin '%s'", resolved_plugin_id) self._set_on_demand_error("no-modes") return - - # Prioritize live modes if they exist and have content - live_modes = [m for m in available_plugin_modes if m.endswith('_live')] - other_modes = [m for m in available_plugin_modes if not m.endswith('_live')] - - # Check if live modes have content - live_with_content = [] - for live_mode in live_modes: - plugin_instance = self.plugin_modes.get(live_mode) - if plugin_instance and hasattr(plugin_instance, 'has_live_content'): - try: - if plugin_instance.has_live_content(): - live_with_content.append(live_mode) - except Exception: - pass - - # Build mode list: live modes with content first, then other modes, then live modes without content - if live_with_content: - ordered_modes = live_with_content + other_modes + [m for m in live_modes if m not in live_with_content] - else: - # No live content, skip live modes - ordered_modes = other_modes - - if not ordered_modes: - # Only live modes available but no content - use them anyway - ordered_modes = live_modes - + + ordered_modes = self._apply_on_demand_pin(ordered_modes, resolved_mode, pinned) + self.on_demand_active = True self.on_demand_mode = resolved_mode # Keep for backward compatibility self.on_demand_modes = ordered_modes diff --git a/systemd/README.md b/systemd/README.md index a2408954f..4eef658bd 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -19,12 +19,35 @@ This directory contains systemd service unit files for LEDMatrix services. - Automatically enables/disables access point mode - Uses `scripts/utils/wifi_monitor_daemon.py` +- **`ledmatrix-dns-fix.service`** - DNS single-request fix (optional) + - Re-applies `options single-request` to the resolver on every boot, + because whatever manages `resolv.conf` regenerates it and drops the + option again + - Works around glibc's parallel A/AAAA lookup stalling ~5s per name on + routers that answer only the A query, which makes any plugin calling an + external API slow or (for Starlark apps, which have a render timeout) + fail outright + - Uses `scripts/utils/apply_dns_single_request.sh` + - Install only if external API calls are timing out; it is not part of a + normal install + +- **`ledmatrix-mqtt-bridge.service`** - Home Assistant MQTT bridge (optional) + - Exposes the display to Home Assistant over MQTT Discovery: force a mode, + stop on-demand, toggle power, set brightness + - Uses `integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py`, which drives the + web API rather than the display directly + - Needs `integrations/mqtt_bridge/bridge_config.json`; see that directory's + README + ## Installation These service files are installed by the installation scripts in `scripts/install/`: - `install_service.sh` installs `ledmatrix.service` - `install_web_service.sh` installs `ledmatrix-web.service` - `install_wifi_monitor.sh` installs `ledmatrix-wifi-monitor.service` +- `install_dns_fix.sh` installs `ledmatrix-dns-fix.service` (opt-in, not run + by the normal installer) +- `install_mqtt_bridge.sh` installs `ledmatrix-mqtt-bridge.service` (opt-in) ## Manual Installation diff --git a/systemd/ledmatrix-dns-fix.service b/systemd/ledmatrix-dns-fix.service new file mode 100644 index 000000000..b48fa2efa --- /dev/null +++ b/systemd/ledmatrix-dns-fix.service @@ -0,0 +1,16 @@ +[Unit] +Description=LED Matrix DNS single-request fix (works around slow A/AAAA lookups) +After=network-online.target NetworkManager.service +Wants=network-online.target +Before=ledmatrix.service + +[Service] +Type=oneshot +ExecStart=__PROJECT_ROOT_DIR__/scripts/utils/apply_dns_single_request.sh +RemainAfterExit=yes +StandardOutput=journal +StandardError=journal +SyslogIdentifier=ledmatrix-dns-fix + +[Install] +WantedBy=multi-user.target diff --git a/systemd/ledmatrix-mqtt-bridge.service b/systemd/ledmatrix-mqtt-bridge.service new file mode 100644 index 000000000..c50370bb9 --- /dev/null +++ b/systemd/ledmatrix-mqtt-bridge.service @@ -0,0 +1,18 @@ +[Unit] +Description=LED Matrix Home Assistant MQTT Bridge +After=network-online.target ledmatrix-web.service +Wants=network-online.target + +[Service] +Type=simple +User=root +WorkingDirectory=__PROJECT_ROOT_DIR__ +ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py +Restart=on-failure +RestartSec=10 +StandardOutput=journal +StandardError=journal +SyslogIdentifier=ledmatrix-mqtt-bridge + +[Install] +WantedBy=multi-user.target diff --git a/test/test_api_v3_display_modes.py b/test/test_api_v3_display_modes.py new file mode 100644 index 000000000..91b3f7906 --- /dev/null +++ b/test/test_api_v3_display_modes.py @@ -0,0 +1,104 @@ +"""GET /api/v3/display/modes -- the list of modes an on-demand request can name. + +Anything driving the display from outside the web UI needs two things that no +existing endpoint gave it: the set of display modes, and which plugin owns each +one. /plugins/installed carries neither, so callers read every plugin's +manifest.json off disk and reimplemented PluginManager's own fallbacks. +""" + +from unittest.mock import MagicMock + +import pytest + +from test._api_v3_test_helpers import ( # noqa: F401 - fixtures + api_v3_client, api_v3_module, build_app, +) + + +MANIFESTS = { + 'clock-simple': {'name': 'Simple Clock', 'display_modes': ['clock-simple']}, + 'football-scoreboard': { + 'name': 'Football Scoreboard', + 'display_modes': ['nfl_live', 'nfl_recent', 'nfl_upcoming'], + }, + 'ledmatrix-weather': {'name': 'Weather', 'display_modes': ['weather']}, +} + +CONFIG = { + 'clock-simple': {'enabled': True}, + 'football-scoreboard': {'enabled': True}, + 'ledmatrix-weather': {'enabled': False}, +} + + +@pytest.fixture +def client(api_v3_module, api_v3_client): + pm = api_v3_module.api_v3.plugin_manager + pm.plugin_manifests = MANIFESTS + pm.discover_plugins = MagicMock(return_value=list(MANIFESTS)) + pm.get_plugin_display_modes = MagicMock( + side_effect=lambda pid: MANIFESTS[pid]['display_modes']) + api_v3_module.api_v3.config_manager.load_config = MagicMock(return_value=CONFIG) + return api_v3_client + + +def _modes(response): + return {m['mode']: m for m in response.get_json()['data']['modes']} + + +class TestTheModeListing: + def test_enabled_plugins_contribute_every_mode(self, client): + modes = _modes(client.get('/api/v3/display/modes')) + assert set(modes) == {'clock-simple', 'nfl_live', 'nfl_recent', 'nfl_upcoming'} + + def test_each_mode_names_its_plugin(self, client): + """on-demand/start's find_plugin_for_mode fallback cannot see generated + modes, so the caller has to send plugin_id -- it must come from here.""" + modes = _modes(client.get('/api/v3/display/modes')) + assert modes['nfl_live']['plugin_id'] == 'football-scoreboard' + + def test_disabled_plugins_are_left_out_by_default(self, client): + modes = _modes(client.get('/api/v3/display/modes')) + assert 'weather' not in modes + + def test_disabled_plugins_can_be_asked_for(self, client): + """They are still valid on-demand targets: the controller enables them + for the duration of the request.""" + modes = _modes(client.get('/api/v3/display/modes?include_disabled=1')) + assert modes['weather']['enabled'] is False + + def test_a_single_mode_plugin_is_labelled_with_its_own_name(self, client): + modes = _modes(client.get('/api/v3/display/modes')) + assert modes['clock-simple']['name'] == 'Simple Clock' + + def test_a_multi_mode_plugin_labels_each_mode_distinctly(self, client): + """There is no per-mode name anywhere, and three modes all called + "Football Scoreboard" are not a usable dropdown.""" + modes = _modes(client.get('/api/v3/display/modes')) + assert modes['nfl_live']['name'] == 'nfl_live' + assert modes['nfl_live']['plugin_name'] == 'Football Scoreboard' + + +class TestItWorksForACallerThatNeverOpensTheDashboard: + def test_discovery_is_triggered(self, client, api_v3_module): + """Discovery is lazy and normally runs because a person loaded the + dashboard; a bridge or script would otherwise get an empty list.""" + client.get('/api/v3/display/modes') + api_v3_module.api_v3.plugin_manager.discover_plugins.assert_called_once() + + def test_no_plugin_manager_is_a_clean_error(self, api_v3_module, api_v3_client): + api_v3_module.api_v3.plugin_manager = None + response = api_v3_client.get('/api/v3/display/modes') + assert response.status_code == 500 + assert response.get_json()['status'] == 'error' + + def test_a_plugin_with_no_declared_modes_still_appears(self, client, api_v3_module): + """Its mode is its own id -- the same fallback the controller uses.""" + pm = api_v3_module.api_v3.plugin_manager + pm.plugin_manifests = {'starlark-apps': {'name': 'Starlark Apps', 'display_modes': []}} + pm.get_plugin_display_modes = MagicMock(return_value=[]) + api_v3_module.api_v3.config_manager.load_config = MagicMock( + return_value={'starlark-apps': {'enabled': True}}) + + modes = _modes(client.get('/api/v3/display/modes')) + assert modes['starlark-apps']['plugin_id'] == 'starlark-apps' diff --git a/test/test_mqtt_bridge.py b/test/test_mqtt_bridge.py new file mode 100644 index 000000000..9840e34be --- /dev/null +++ b/test/test_mqtt_bridge.py @@ -0,0 +1,266 @@ +"""The Home Assistant MQTT bridge, without a broker or a matrix. + +The bridge is a translation layer: one MQTT payload in, one api_v3 call out. +Everything worth pinning is on that path -- attaching the plugin_id a mode +needs, rejecting values HA can produce but the API cannot take, and the +discovery configs HA reads once and caches -- so it is all reachable with a +fake API client and a pure function. +""" + +import importlib.util +import json +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +BRIDGE_PATH = (Path(__file__).resolve().parent.parent + / "integrations" / "mqtt_bridge" / "ledmatrix_mqtt_bridge.py") + + +@pytest.fixture(scope="module") +def bridge_module(): + if not BRIDGE_PATH.exists(): + pytest.skip("mqtt bridge is not present") + try: + spec = importlib.util.spec_from_file_location("ledmatrix_mqtt_bridge", BRIDGE_PATH) + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + except ImportError as e: + pytest.skip(f"mqtt bridge dependencies are not installed here: {e}") + finally: + sys.modules.pop("ledmatrix_mqtt_bridge", None) + + +MODES = [ + {"mode": "clock-simple", "plugin_id": "clock-simple", + "plugin_name": "Simple Clock", "name": "Simple Clock", "enabled": True}, + {"mode": "nfl_live", "plugin_id": "football-scoreboard", + "plugin_name": "Football Scoreboard", "name": "nfl_live", "enabled": True}, + {"mode": "aquarium", "plugin_id": "starlark-apps", + "plugin_name": "Starlark Apps", "name": "aquarium", "enabled": True}, +] + + +@pytest.fixture +def client(): + """A stand-in for LEDMatrixClient that records what it was asked to do.""" + fake = MagicMock() + fake.list_modes.return_value = list(MODES) + fake.start_on_demand.return_value = {"request_id": "r1"} + fake.stop_on_demand.return_value = {} + fake.set_power.return_value = {} + fake.set_brightness.return_value = {} + fake.get_brightness.return_value = 90 + fake.display_status.return_value = { + "service": {"active": True}, + "state": {"active": True, "mode": "nfl_live"}, + } + return fake + + +@pytest.fixture +def handler(bridge_module, client): + h = bridge_module.CommandHandler(client) + h.refresh_modes() + return h + + +class TestDisplayCommands: + def test_a_mode_is_forced_on_demand(self, handler, client): + result = handler.handle({"action": "display", "mode": "nfl_live"}) + assert result["status"] == "success" + assert client.start_on_demand.call_args.kwargs["mode"] == "nfl_live" + + def test_the_owning_plugin_is_sent_with_the_mode(self, handler, client): + """on-demand/start's find_plugin_for_mode fallback cannot resolve a + generated mode, so an omitted plugin_id 404s for every Starlark app.""" + handler.handle({"action": "display", "mode": "aquarium"}) + assert client.start_on_demand.call_args.kwargs["plugin_id"] == "starlark-apps" + + def test_a_home_assistant_label_resolves_to_its_mode(self, handler, client): + """The select entity shows labels, so that is what comes back.""" + handler.handle({"action": "display", "mode": "Simple Clock"}) + assert client.start_on_demand.call_args.kwargs["mode"] == "clock-simple" + + def test_an_unknown_mode_refreshes_before_giving_up(self, handler, client): + """A plugin installed since the last refresh is the usual reason.""" + client.list_modes.return_value = MODES + [ + {"mode": "new_app", "plugin_id": "starlark-apps", "name": "new_app"}] + handler.handle({"action": "display", "mode": "new_app"}) + assert client.start_on_demand.call_args.kwargs["plugin_id"] == "starlark-apps" + + def test_an_explicit_plugin_id_is_respected(self, handler, client): + handler.handle({"action": "display", "mode": "x", "plugin_id": "custom"}) + assert client.start_on_demand.call_args.kwargs["plugin_id"] == "custom" + + def test_duration_and_pinned_are_passed_through(self, handler, client): + handler.handle({"action": "display", "mode": "aquarium", + "duration": 300, "pinned": True}) + kwargs = client.start_on_demand.call_args.kwargs + assert kwargs["duration"] == 300 and kwargs["pinned"] is True + + def test_display_with_nothing_to_show_is_rejected(self, handler, client): + result = handler.handle({"action": "display"}) + assert result["status"] == "error" + client.start_on_demand.assert_not_called() + + def test_stop_returns_to_normal_rotation(self, handler, client): + assert handler.handle({"action": "stop_display"})["status"] == "success" + client.stop_on_demand.assert_called_once() + + +class TestPowerAndBrightness: + @pytest.mark.parametrize("state,expected", [("on", True), ("OFF", False), (" On ", True)]) + def test_power_states_are_accepted(self, handler, client, state, expected): + assert handler.handle({"action": "power", "state": state})["status"] == "success" + client.set_power.assert_called_with(expected) + + def test_an_unknown_power_state_is_rejected(self, handler, client): + assert handler.handle({"action": "power", "state": "maybe"})["status"] == "error" + client.set_power.assert_not_called() + + def test_brightness_is_applied(self, handler, client): + assert handler.handle({"action": "brightness", "value": 75})["status"] == "success" + client.set_brightness.assert_called_with(75) + + def test_a_float_from_home_assistant_is_accepted(self, handler, client): + """The number entity publishes "75.0"; int() would raise on that.""" + handler.handle({"action": "brightness", "value": "75.0"}) + client.set_brightness.assert_called_with(75) + + @pytest.mark.parametrize("value", [-1, 101, "bright", None]) + def test_out_of_range_brightness_never_reaches_the_api(self, handler, client, value): + assert handler.handle({"action": "brightness", "value": value})["status"] == "error" + client.set_brightness.assert_not_called() + + +class TestFailuresAreReportedNotRaised: + """A failed command must publish an error, not kill the MQTT loop.""" + + def test_an_api_error_becomes_an_error_result(self, handler, client): + client.start_on_demand.side_effect = RuntimeError("Display service is not running") + result = handler.handle({"action": "display", "mode": "nfl_live"}) + assert result["status"] == "error" + assert "not running" in result["message"] + + def test_an_unknown_action_lists_the_known_ones(self, handler): + result = handler.handle({"action": "explode"}) + assert result["status"] == "error" + assert "brightness" in result["message"] + + def test_a_missing_action_is_an_error(self, handler): + assert handler.handle({})["status"] == "error" + + +class TestDiscoveryPayloads: + """HA reads these once and caches them; a wrong shape is a dead entity.""" + + @pytest.fixture + def messages(self, bridge_module): + return bridge_module.discovery_messages( + "ledmatrix/command", "ledmatrix/command/state", + "ledmatrix/command/availability", ["Simple Clock", "nfl_live"]) + + def test_every_entity_is_published(self, messages): + components = {m["topic"].split("/")[1] for m in messages} + assert components == {"select", "button", "switch", "number"} + + def test_each_entity_has_a_stable_unique_id(self, messages): + """Without one HA cannot let a user rename or reassign the entity.""" + ids = [m["payload"]["unique_id"] for m in messages] + assert len(ids) == len(set(ids)) and all(ids) + + def test_the_select_offers_the_modes_it_was_given(self, messages): + select = next(m for m in messages if "/select/" in m["topic"]) + assert select["payload"]["options"] == ["Simple Clock", "nfl_live"] + + def test_every_entity_shares_the_availability_topic(self, messages): + """It is also the bridge's last will, so HA greys the controls out + when the bridge dies rather than leaving them silently inert.""" + assert all(m["payload"]["availability_topic"] == "ledmatrix/command/availability" + for m in messages) + + def test_every_entity_belongs_to_one_device(self, messages): + assert all(m["payload"]["device"]["identifiers"] == ["ledmatrix"] for m in messages) + + def test_the_command_templates_are_valid_json_the_handler_accepts(self, messages, handler): + """A template that renders malformed JSON fails only at runtime, in HA.""" + select = next(m for m in messages if "/select/" in m["topic"]) + rendered = select["payload"]["command_template"].replace("{{ value }}", "nfl_live") + assert handler.handle(json.loads(rendered))["status"] == "success" + + number = next(m for m in messages if "/number/" in m["topic"]) + rendered = number["payload"]["command_template"].replace("{{ value }}", "40") + assert handler.handle(json.loads(rendered))["status"] == "success" + + def test_the_switch_payloads_are_valid_json_the_handler_accepts(self, messages, handler): + switch = next(m for m in messages if "/switch/" in m["topic"]) + for key in ("payload_on", "payload_off"): + assert handler.handle(json.loads(switch["payload"][key]))["status"] == "success" + + def test_the_button_payload_is_valid_json_the_handler_accepts(self, messages, handler): + button = next(m for m in messages if "/button/" in m["topic"]) + assert handler.handle(json.loads(button["payload"]["payload_press"]))["status"] == "success" + + +class TestStateReporting: + def test_state_reflects_the_running_display(self, bridge_module, client): + state = bridge_module.read_state(client) + assert state == {"power": True, "mode": "nfl_live", "brightness": 90} + + def test_normal_rotation_reports_no_forced_mode(self, bridge_module, client): + client.display_status.return_value = { + "service": {"active": True}, "state": {"active": False}} + assert bridge_module.read_state(client)["mode"] is None + + def test_an_unreachable_field_does_not_blank_the_others(self, bridge_module, client): + """A stopped display service still has a brightness worth showing.""" + client.display_status.side_effect = RuntimeError("connection refused") + state = bridge_module.read_state(client) + assert state["brightness"] == 90 and state["power"] is False + + +class TestConfigLoading: + def test_defaults_apply_when_no_file_exists(self, bridge_module, tmp_path): + config = bridge_module.load_config(str(tmp_path / "absent.json")) + assert config["mqtt_topic"] == "ledmatrix/command" + assert config["mqtt_port"] == 1883 + + def test_the_file_overrides_defaults(self, bridge_module, tmp_path): + path = tmp_path / "bridge_config.json" + path.write_text(json.dumps({"mqtt_host": "broker.local", "mqtt_port": "8883"})) + config = bridge_module.load_config(str(path)) + assert config["mqtt_host"] == "broker.local" + assert config["mqtt_port"] == 8883, "a port read from JSON must still be an int" + + def test_the_environment_overrides_the_file(self, bridge_module, tmp_path, monkeypatch): + """So a password need not sit in a file the service user can read.""" + path = tmp_path / "bridge_config.json" + path.write_text(json.dumps({"mqtt_password": "from-file"})) + monkeypatch.setenv("LEDMATRIX_MQTT_MQTT_PASSWORD", "from-env") + assert bridge_module.load_config(str(path))["mqtt_password"] == "from-env" + + def test_the_example_placeholder_password_is_refused(self, bridge_module, tmp_path): + """Copying the example unedited must fail loudly, not silently fail to + authenticate against the broker.""" + path = tmp_path / "bridge_config.json" + path.write_text(json.dumps( + {"mqtt_password": "REPLACE_WITH_YOUR_ACTUAL_MQTT_PASSWORD"})) + with pytest.raises(bridge_module.ConfigError): + bridge_module.load_config(str(path)) + + def test_malformed_json_is_a_clear_error(self, bridge_module, tmp_path): + path = tmp_path / "bridge_config.json" + path.write_text("{not json") + with pytest.raises(bridge_module.ConfigError): + bridge_module.load_config(str(path)) + + def test_a_non_numeric_port_is_a_clear_error(self, bridge_module, tmp_path): + path = tmp_path / "bridge_config.json" + path.write_text(json.dumps({"mqtt_port": "not-a-port"})) + with pytest.raises(bridge_module.ConfigError): + bridge_module.load_config(str(path)) diff --git a/test/test_on_demand_pinning_and_restart.py b/test/test_on_demand_pinning_and_restart.py new file mode 100644 index 000000000..7d32a3202 --- /dev/null +++ b/test/test_on_demand_pinning_and_restart.py @@ -0,0 +1,208 @@ +"""On-demand behaviour that a person can ask for but the controller ignored. + +Three separate gaps, all reachable from the web UI's force-display dialog: + + * `pinned` was accepted by the API, stored on the controller and published + back in the status payload, but never narrowed the rotation -- a pinned + request still cycled every mode its plugin owns; + * restarting while on-demand was active loaded *only* the on-demand plugin, + so normal rotation had nothing to return to for the life of the process; + * a stop request was exempt from the duplicate guards on purpose and was + never removed from the mailbox, so it was re-processed on every poll + forever. +""" + +from unittest.mock import MagicMock + +import pytest + + +def _plugin_with_modes(controller, plugin_id, modes): + """Register `modes` as loaded modes belonging to `plugin_id`.""" + controller.plugin_display_modes[plugin_id] = list(modes) + for mode in modes: + controller.plugin_modes[mode] = MagicMock(spec=[]) + controller.mode_to_plugin_id[mode] = plugin_id + + +class TestPinnedNarrowsTheRotation: + """A pinned request shows the one mode that was asked for. + + Unpinned stays the default: a sports plugin's modes are views of one + subject, so rotating them is right. A Starlark plugin's modes are + unrelated widgets, so it is not. + """ + + MODES = ['app_a', 'app_b', 'app_c'] + + def _activate(self, controller, pinned): + _plugin_with_modes(controller, 'starlark-apps', self.MODES) + controller.available_modes = list(self.MODES) + controller._activate_on_demand({ + 'request_id': 'r1', + 'action': 'start', + 'plugin_id': 'starlark-apps', + 'mode': 'app_b', + 'pinned': pinned, + }) + + def test_pinned_rotation_holds_the_requested_mode(self, test_display_controller): + c = test_display_controller + self._activate(c, pinned=True) + assert c.on_demand_modes == ['app_b'] + + def test_unpinned_still_rotates_the_whole_plugin(self, test_display_controller): + c = test_display_controller + self._activate(c, pinned=False) + assert set(c.on_demand_modes) == set(self.MODES) + + def test_pinned_still_starts_on_the_requested_mode(self, test_display_controller): + c = test_display_controller + self._activate(c, pinned=True) + assert c.current_display_mode == 'app_b' + + def test_the_pin_is_recorded_for_the_status_payload(self, test_display_controller): + c = test_display_controller + self._activate(c, pinned=True) + assert c.on_demand_pinned is True + + def test_an_unresolvable_pin_does_not_empty_the_rotation(self, test_display_controller): + """A pin naming a mode outside the plugin must not leave nothing to show.""" + c = test_display_controller + _plugin_with_modes(c, 'starlark-apps', self.MODES) + assert c._apply_on_demand_pin(list(self.MODES), 'not_a_mode', True) == self.MODES + + +class TestPinSurvivesARestart: + """The pin is part of the request being resumed, not a per-session flag.""" + + def test_restored_pinned_state_narrows_the_rotation(self, test_display_controller): + c = test_display_controller + _plugin_with_modes(c, 'starlark-apps', ['app_a', 'app_b', 'app_c']) + c.on_demand_active = True + c.on_demand_plugin_id = 'starlark-apps' + c.on_demand_mode = 'app_c' + c.on_demand_pinned = True + + c._populate_on_demand_modes_from_plugin() + assert c.on_demand_modes == ['app_c'] + + def test_restored_unpinned_state_keeps_every_mode(self, test_display_controller): + c = test_display_controller + _plugin_with_modes(c, 'starlark-apps', ['app_a', 'app_b', 'app_c']) + c.on_demand_active = True + c.on_demand_plugin_id = 'starlark-apps' + c.on_demand_mode = 'app_c' + c.on_demand_pinned = False + + c._populate_on_demand_modes_from_plugin() + assert set(c.on_demand_modes) == {'app_a', 'app_b', 'app_c'} + + +class TestRestartDoesNotStarveTheOtherPlugins: + """Restarting mid-on-demand used to load only the on-demand plugin. + + Restarts during an on-demand session are routine -- it is how an update or + a config change is applied -- and the panel came back cycling one plugin's + modes and nothing else until the on-demand cache was cleared by hand. + """ + + DISCOVERED = ['clock', 'weather', 'starlark-apps', 'disabled-one'] + + @pytest.fixture + def controller(self, test_display_controller): + c = test_display_controller + c.config.update({ + 'clock': {'enabled': True}, + 'weather': {'enabled': True}, + 'starlark-apps': {'enabled': True}, + 'disabled-one': {'enabled': False}, + }) + return c + + def test_every_enabled_plugin_still_loads(self, controller): + selected = controller._select_startup_plugins( + self.DISCOVERED, {'plugin_id': 'starlark-apps', 'mode': 'app_a'}) + assert set(selected) == {'clock', 'weather', 'starlark-apps'} + + def test_disabled_plugins_are_still_left_out(self, controller): + selected = controller._select_startup_plugins( + self.DISCOVERED, {'plugin_id': 'starlark-apps', 'mode': 'app_a'}) + assert 'disabled-one' not in selected + + def test_the_on_demand_state_is_still_restored(self, controller): + controller._select_startup_plugins( + self.DISCOVERED, + {'plugin_id': 'starlark-apps', 'mode': 'app_a', 'pinned': True}) + assert controller.on_demand_active is True + assert controller.on_demand_plugin_id == 'starlark-apps' + assert controller.on_demand_mode == 'app_a' + assert controller.on_demand_pinned is True + + def test_a_disabled_on_demand_plugin_is_enabled_and_loaded(self, controller): + """Otherwise the mode being resumed has nothing behind it.""" + selected = controller._select_startup_plugins( + self.DISCOVERED, {'plugin_id': 'disabled-one', 'mode': 'x'}) + assert 'disabled-one' in selected + assert controller.config['disabled-one']['enabled'] is True + + def test_an_unknown_on_demand_plugin_falls_back_to_normal(self, controller): + selected = controller._select_startup_plugins( + self.DISCOVERED, {'plugin_id': 'uninstalled', 'mode': 'x'}) + assert set(selected) == {'clock', 'weather', 'starlark-apps'} + assert controller.on_demand_active is False + + def test_no_on_demand_config_is_a_normal_startup(self, controller): + selected = controller._select_startup_plugins(self.DISCOVERED, None) + assert set(selected) == {'clock', 'weather', 'starlark-apps'} + assert controller.on_demand_active is False + + +class TestStopRequestsAreConsumed: + """A stop request is exempt from the duplicate guards, so the mailbox + delete is the only thing that ends it.""" + + STOP = {'request_id': 'S1', 'action': 'stop'} + + def _arrange(self, controller, active): + controller.on_demand_active = active + controller.on_demand_status = 'active' if active else 'idle' + controller._last_on_demand_poll = None + controller.cache_manager.get = MagicMock( + side_effect=lambda key, *a, **kw: + self.STOP if key == 'display_on_demand_request' else None) + controller.cache_manager.set = MagicMock() + controller.cache_manager.delete = MagicMock() + controller._clear_on_demand = MagicMock() + + def test_a_handled_stop_is_removed_from_the_mailbox(self, test_display_controller): + c = test_display_controller + self._arrange(c, active=True) + c._poll_on_demand_requests() + c.cache_manager.delete.assert_called_once_with('display_on_demand_request') + + def test_a_stop_arriving_while_idle_is_also_removed(self, test_display_controller): + """Otherwise a stop sent to an idle display re-fires forever.""" + c = test_display_controller + self._arrange(c, active=False) + c._poll_on_demand_requests() + c.cache_manager.delete.assert_called_once_with('display_on_demand_request') + + def test_the_stop_is_still_acted_on(self, test_display_controller): + c = test_display_controller + self._arrange(c, active=True) + c._poll_on_demand_requests() + c._clear_on_demand.assert_called_once_with(reason='requested-stop') + + def test_a_start_racing_in_behind_a_stop_is_not_discarded(self, test_display_controller): + """The compare-before-delete applies to stops too.""" + c = test_display_controller + self._arrange(c, active=True) + newer = {'request_id': 'S2', 'action': 'start', 'plugin_id': 'p', 'mode': 'm'} + reads = iter([self.STOP, newer]) + c.cache_manager.get = MagicMock( + side_effect=lambda key, *a, **kw: + next(reads, newer) if key == 'display_on_demand_request' else None) + + c._poll_on_demand_requests() + assert c.cache_manager.delete.call_count == 0 diff --git a/test/test_pixlet_renderer_contract.py b/test/test_pixlet_renderer_contract.py new file mode 100644 index 000000000..2aede05be --- /dev/null +++ b/test/test_pixlet_renderer_contract.py @@ -0,0 +1,210 @@ +"""starlark-apps PixletRenderer: what reaches Pixlet, and what counts as a render. + +Three things a Starlark app can do that the renderer got wrong: + + * put a "|" in a config value -- a shell metacharacter filter dropped the + whole key, though the command is a list and no shell is involved; + * render nothing -- Pixlet exits 0 and writes a 0-byte file, which was + reported as a successful render; + * compute its schema at runtime -- the source parser can only read option + lists that are written out literally, so a dropdown fed by a live API call + came back empty. +""" + +import importlib.util +import json +import subprocess +import sys +from pathlib import Path +from unittest.mock import patch + +import pytest + +PLUGIN_DIR = Path(__file__).resolve().parent.parent / "plugin-repos" / "starlark-apps" + + +@pytest.fixture(scope="module") +def renderer_module(): + if not PLUGIN_DIR.exists(): + pytest.skip("starlark-apps plugin is not checked out") + sys.path.insert(0, str(PLUGIN_DIR)) + try: + spec = importlib.util.spec_from_file_location( + "pixlet_renderer_under_test", PLUGIN_DIR / "pixlet_renderer.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + except Exception as e: # noqa: BLE001 - optional deps may be absent + pytest.skip(f"pixlet_renderer is not importable here: {e}") + finally: + sys.path.remove(str(PLUGIN_DIR)) + + +@pytest.fixture +def renderer(renderer_module): + """A renderer with a known binary and __init__'s binary search bypassed.""" + r = renderer_module.PixletRenderer.__new__(renderer_module.PixletRenderer) + r.timeout = 30 + r.pixlet_binary = "/usr/local/bin/pixlet" + return r + + +def _completed(returncode=0, stdout="", stderr=""): + return subprocess.CompletedProcess(args=[], returncode=returncode, + stdout=stdout, stderr=stderr) + + +class TestConfigValuesReachPixlet: + """cmd is a list and there is no shell=True, so nothing here is ever + interpreted by a shell -- the filter is defence in depth, not a boundary.""" + + def _args_for(self, renderer, tmp_path, config): + star = tmp_path / "app.star" + star.write_text("# app", encoding="utf-8") + out = tmp_path / "out.webp" + + def fake_run(cmd, **kw): + out.write_bytes(b"webp-bytes") + fake_run.cmd = cmd + return _completed() + + with patch.object(subprocess, "run", side_effect=fake_run): + renderer.render(str(tmp_path / "app.star"), str(out), config=config) + return fake_run.cmd + + def test_a_pipe_in_a_value_is_passed_through(self, renderer, tmp_path): + """Real apps use "|" as a separator inside one config value.""" + args = self._args_for(renderer, tmp_path, {"sign_id": "I-476 North|175659"}) + assert "sign_id=I-476 North|175659" in args + + def test_a_dropped_value_does_not_take_the_key_with_it(self, renderer, tmp_path): + args = self._args_for(renderer, tmp_path, {"sign_id": "I-476 North|175659"}) + assert any(a.startswith("sign_id=") for a in args) + + @pytest.mark.parametrize("value", [ + "$(rm -rf /)", + "`whoami`", + "a;b", + "a&b", + "a>b", + "a
-

${escapeHtml(app.name || app.id)}

+ +

${escapeHtml(app.name || app.id)}

Starlark ${installed ? 'Installed' : ''}
From 2f7b774d1cedd2f48968883f0343135ef30c3203 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 7 Sep 2026 16:57:44 -0400 Subject: [PATCH 2/3] fix(mqtt_bridge): the five issues Codacy flagged on this branch All in the new bridge, all real: * requests floor was 2.31.0, which carries CVE-2024-35195, CVE-2024-47081 and CVE-2026-25645. Raised to >=2.33.0,<3.0.0, which is what the project's own requirements.txt already pins. * `import time` was never used. * `"mqtt_password": None` in DEFAULTS read as a hardcoded credential. It is the "no password configured" default; marked nosec B105, the convention used elsewhere in the repo. Also dropped an unused `build_app` from the display-modes test imports. Co-Authored-By: Claude Opus 5 --- integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py | 3 +-- integrations/mqtt_bridge/requirements.txt | 7 +++++-- test/test_api_v3_display_modes.py | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py index dca657f3d..b6ea48d9b 100644 --- a/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py +++ b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py @@ -41,7 +41,6 @@ import signal import sys import threading -import time from typing import Any, Callable, Dict, List, Optional import requests @@ -61,7 +60,7 @@ "mqtt_host": "localhost", "mqtt_port": 1883, "mqtt_username": None, - "mqtt_password": None, + "mqtt_password": None, # nosec B105 - "no password configured", not a credential "mqtt_client_id": "ledmatrix-mqtt-bridge", "mqtt_topic": "ledmatrix/command", "mqtt_tls": False, diff --git a/integrations/mqtt_bridge/requirements.txt b/integrations/mqtt_bridge/requirements.txt index 7bd6cac8d..0e5d99d03 100644 --- a/integrations/mqtt_bridge/requirements.txt +++ b/integrations/mqtt_bridge/requirements.txt @@ -1,2 +1,5 @@ -paho-mqtt>=2.0.0 -requests>=2.31.0 +# Floors are security floors, not API floors. requests < 2.33.0 carries +# CVE-2024-35195, CVE-2024-47081 and CVE-2026-25645; matches the pin in the +# project's own requirements.txt. +paho-mqtt>=2.0.0,<3.0.0 +requests>=2.33.0,<3.0.0 diff --git a/test/test_api_v3_display_modes.py b/test/test_api_v3_display_modes.py index 91b3f7906..2977f6896 100644 --- a/test/test_api_v3_display_modes.py +++ b/test/test_api_v3_display_modes.py @@ -11,7 +11,7 @@ import pytest from test._api_v3_test_helpers import ( # noqa: F401 - fixtures - api_v3_client, api_v3_module, build_app, + api_v3_client, api_v3_module, ) From a0d2140d9ff7b4223c913d01f5cbb3a854f7fc50 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 7 Sep 2026 17:17:06 -0400 Subject: [PATCH 3/3] fix: the review findings on this PR Nine of CodeRabbit's ten, plus the CodeQL alert. The tenth is wrong and is answered below. **One bad config section blanked the whole mode list.** `/display/modes` read `full_config.get(plugin_id, {}).get('enabled')`, so a non-dict under a plugin id -- a shape DisplayController already guards, so it happens -- raised AttributeError mid-loop and answered 500 with no modes at all. Every MQTT bridge entity is built from that list. Now skipped with a warning. **The DNS scripts reported success they had not earned.** Three separate paths: `resolvconf -u` failing was swallowed by `|| true`; the systemd-resolved branch exited 0 without applying anything, so the oneshot unit recorded success while the workaround was inactive; and the installer's `|| echo` turned a failed start into "installation complete." with exit 0. All three now fail loudly. `single-request` is a glibc resolv.conf option with no resolved.conf equivalent, so on those hosts the honest answer is that it cannot be applied. A NetworkManager-generated resolv.conf is regenerated on connection changes, not only at boot, and the unit is oneshot with RemainAfterExit -- so the option can vanish mid-boot with nothing to put it back. Now detected and stated plainly rather than implied to be permanent. **`Before=` does not order a manual restart.** It only orders units already in the same transaction, so `systemctl restart ledmatrix` could bypass the fix. install_dns_fix.sh now writes a ledmatrix.service drop-in with Wants= and After=. Wants=, not Requires=: a DNS workaround failing should not stop the display. **The Pixlet editor's `--lan` is gone.** `pixlet serve` has no authentication, and a printed warning is not access control. Loopback only, with the SSH port-forward in the header where the flag used to be documented -- SSH does the authenticating and nothing is left listening. **The MQTT example config now defaults to TLS** on 8883. The installer copies it verbatim, and without TLS the broker password and every command cross the network in cleartext. A plaintext broker is still supported and documented, and the bridge warns once at startup when a password is configured without TLS. **Not taken: "the upstream Pixlet CLI has no `schema` subcommand."** Upstream tidbyt/pixlet has none, but `scripts/download_pixlet.sh` installs `tronbyt/pixlet`, whose `cmd/schema.go` is `schema [PATH]` -> JSON on stdout, built on `runtime.NewAppletFromPath`, so it does execute `get_schema()`. That is exactly what extract_schema_via_pixlet calls. A binary without the subcommand exits non-zero and falls back to the source parser, which is already covered by a test. **CodeQL stack-trace exposure: not taken either.** I removed `details` first and that broke test_web_error_detail.py::test_no_api_v3_handler_discards_its_exception, which enforces `describe_exception` across all ~75 handlers -- written because a device with failing storage answered "see logs for details" from the log viewer itself. describe_exception redacts credentials; the trade-off is the project's and is already made. Restored, with the reasoning in a comment. 11 new tests. Whole suite: no new failures against main, 4127 passed. Co-Authored-By: Claude Opus 5 --- integrations/mqtt_bridge/README.md | 10 ++- .../mqtt_bridge/bridge_config.example.json | 4 +- .../mqtt_bridge/ledmatrix_mqtt_bridge.py | 19 ++++++ scripts/install/install_dns_fix.sh | 31 ++++++++- scripts/utils/README.md | 6 +- scripts/utils/apply_dns_single_request.sh | 51 +++++++++++---- scripts/utils/pixlet_config_editor.sh | 33 ++++------ test/test_api_v3_display_modes.py | 53 +++++++++++++++ test/test_mqtt_bridge.py | 65 ++++++++++++++++++- web_interface/blueprints/api_v3.py | 21 +++++- 10 files changed, 251 insertions(+), 42 deletions(-) diff --git a/integrations/mqtt_bridge/README.md b/integrations/mqtt_bridge/README.md index fed0d564f..29d31b041 100644 --- a/integrations/mqtt_bridge/README.md +++ b/integrations/mqtt_bridge/README.md @@ -76,15 +76,21 @@ Edit the config with your broker details and re-run it. ```json { "mqtt_host": "192.168.1.10", - "mqtt_port": 1883, + "mqtt_port": 8883, "mqtt_username": "ledmatrix", "mqtt_password": null, "mqtt_topic": "ledmatrix/command", - "mqtt_tls": false, + "mqtt_tls": true, "ledmatrix_api_base": "http://localhost:5000" } ``` +**TLS is on by default.** Without it the broker password and every display +command cross the network in cleartext. If your broker only listens on plain +1883 — which the Mosquitto add-on does out of the box — set `"mqtt_tls": false` +and `"mqtt_port": 1883`. The bridge logs a warning at startup when a password +is configured without TLS. + `bridge_config.json` is gitignored. Any key can also be supplied through the environment as `LEDMATRIX_MQTT_` (`LEDMATRIX_MQTT_MQTT_PASSWORD`, say), which keeps a broker password out of a file on disk — put it in a systemd diff --git a/integrations/mqtt_bridge/bridge_config.example.json b/integrations/mqtt_bridge/bridge_config.example.json index 98188cfc3..df9c02148 100644 --- a/integrations/mqtt_bridge/bridge_config.example.json +++ b/integrations/mqtt_bridge/bridge_config.example.json @@ -1,11 +1,11 @@ { "mqtt_host": "192.168.1.10", - "mqtt_port": 1883, + "mqtt_port": 8883, "mqtt_username": "ledmatrix", "mqtt_password": null, "mqtt_client_id": "ledmatrix-mqtt-bridge", "mqtt_topic": "ledmatrix/command", - "mqtt_tls": false, + "mqtt_tls": true, "ledmatrix_api_base": "http://localhost:5000", "request_timeout": 15, "on_demand_duration": null, diff --git a/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py index b6ea48d9b..a2199a637 100644 --- a/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py +++ b/integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py @@ -354,6 +354,23 @@ def discovery_messages(command_topic: str, state_topic: str, availability_topic: ] +def warn_if_cleartext(config: Dict[str, Any]) -> bool: + """Say so, once, when a broker password is going over an unencrypted link. + + The shipped example has TLS on, so reaching here means somebody turned it + off deliberately -- which is legitimate (the Mosquitto add-on is plaintext + on 1883) but should not be silent when there is a password to lose. Returns + whether it warned, so the decision is testable without a broker. + """ + if config.get("mqtt_tls") or not config.get("mqtt_password"): + return False + logger.warning( + 'mqtt_tls is off and a password is set: the broker password and every ' + 'command are sent unencrypted. Set "mqtt_tls": true (port 8883 on most ' + 'brokers) unless this is a trusted, isolated network.') + return True + + def read_state(client: LEDMatrixClient) -> Dict[str, Any]: """The state every entity reads, so HA opens on real values. @@ -471,6 +488,8 @@ def run(self) -> int: if self.config.get("mqtt_tls_insecure"): logger.warning("TLS certificate verification is disabled (mqtt_tls_insecure)") self._mqtt.tls_insecure_set(True) + else: + warn_if_cleartext(self.config) self._mqtt.will_set(self.availability_topic, "offline", qos=1, retain=True) self._mqtt.on_connect = self._on_connect diff --git a/scripts/install/install_dns_fix.sh b/scripts/install/install_dns_fix.sh index 94975f0de..1edf2785e 100644 --- a/scripts/install/install_dns_fix.sh +++ b/scripts/install/install_dns_fix.sh @@ -13,6 +13,7 @@ PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) SERVICE_NAME="ledmatrix-dns-fix" UNIT_SRC="$PROJECT_ROOT_DIR/systemd/$SERVICE_NAME.service" UNIT_DEST="/etc/systemd/system/$SERVICE_NAME.service" +DROPIN_DIR="/etc/systemd/system/ledmatrix.service.d" if [ "$EUID" -eq 0 ]; then SYSTEMCTL_CMD="systemctl" @@ -36,13 +37,37 @@ echo "Installing $UNIT_DEST..." sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ | $SUDO tee "$UNIT_DEST" > /dev/null +# Order ledmatrix.service after the fix. `Before=` in the unit itself only +# orders units already in the same transaction, so a plain +# `systemctl restart ledmatrix` would not wait for it -- and since this fix is +# opt-in, ledmatrix.service cannot carry the dependency in the repo. +# Wants=, not Requires=: a DNS workaround failing should not stop the display. +echo "Installing the ledmatrix.service ordering drop-in..." +$SUDO mkdir -p "$DROPIN_DIR" +printf '[Unit]\nWants=%s.service\nAfter=%s.service\n' "$SERVICE_NAME" "$SERVICE_NAME" \ + | $SUDO tee "$DROPIN_DIR/10-dns-fix.conf" > /dev/null + $SYSTEMCTL_CMD daemon-reload $SYSTEMCTL_CMD enable "$SERVICE_NAME.service" -$SYSTEMCTL_CMD start "$SERVICE_NAME.service" || echo "⚠ Failed to start service (will retry on reboot)" + +# Do not mask a failure here. The unit exits non-zero when it could not apply +# the option -- a systemd-resolved host, an unwritable resolv.conf, a failed +# `resolvconf -u` -- and reporting "installation complete" over that would +# leave the operator believing a workaround is active when it is not. +START_STATUS=0 +$SYSTEMCTL_CMD start "$SERVICE_NAME.service" || START_STATUS=$? echo "" if grep -qs "^options single-request$" /etc/resolv.conf; then echo "✓ 'options single-request' is active in /etc/resolv.conf" +elif [ "$START_STATUS" -ne 0 ]; then + echo "✗ The DNS fix could not be applied on this host." + echo " The service reported why:" + echo " journalctl -u $SERVICE_NAME -n 20" + echo "" + echo " The unit is installed and will try again on the next boot. Nothing" + echo " else about your install has changed." + exit "$START_STATUS" else echo "⚠ 'options single-request' is not in /etc/resolv.conf yet." echo " Check what the service reported:" @@ -55,5 +80,7 @@ echo "" echo "Useful commands:" echo " sudo systemctl status $SERVICE_NAME # Check status" echo " sudo journalctl -u $SERVICE_NAME -n 50 # View logs" -echo " sudo systemctl disable --now $SERVICE_NAME # Undo (edit /etc/resolv.conf to remove the line)" +echo " sudo systemctl disable --now $SERVICE_NAME # Undo the service" +echo " sudo rm $DROPIN_DIR/10-dns-fix.conf # Undo the ordering drop-in" +echo " # then remove the 'options single-request' line from /etc/resolv.conf" echo "" diff --git a/scripts/utils/README.md b/scripts/utils/README.md index 5c9d6843f..61bd2ac48 100644 --- a/scripts/utils/README.md +++ b/scripts/utils/README.md @@ -35,7 +35,6 @@ render preview, cascading dropdowns -- rather than the LEDMatrix one. ```bash ./scripts/utils/pixlet_config_editor.sh # list installed apps ./scripts/utils/pixlet_config_editor.sh penndot_signs # edit, on localhost:8080 -./scripts/utils/pixlet_config_editor.sh penndot_signs --lan # reachable from the LAN ``` Deliberately not a service. It stops the display for the length of the @@ -43,7 +42,10 @@ session and `pixlet serve` listens with no authentication, so it should only be running while you are actually editing. It backs the config up first and restarts the display on exit, however it exits. -From another machine, forward the port rather than using `--lan`: +It binds loopback only, with no flag to change that: anything that can reach +`pixlet serve` can rewrite the app's config, and a printed warning is not +access control. To edit from another machine, forward the port -- SSH does the +authenticating and nothing is left listening on the LAN: ```bash ssh -L 8080:localhost:8080 pi@ledpi.local diff --git a/scripts/utils/apply_dns_single_request.sh b/scripts/utils/apply_dns_single_request.sh index c8dbb1c1d..1c94ad886 100644 --- a/scripts/utils/apply_dns_single_request.sh +++ b/scripts/utils/apply_dns_single_request.sh @@ -41,18 +41,29 @@ if [ -d "$(dirname "$RESOLVCONF_TAIL")" ]; then echo "$OPTION" >> "$RESOLVCONF_TAIL" log "added to $RESOLVCONF_TAIL" fi - command -v resolvconf >/dev/null 2>&1 && resolvconf -u || true + # Only a missing resolvconf is ignorable. If it is present and the + # regeneration fails, /etc/resolv.conf still lacks the option, and + # reporting success would be a lie. + if command -v resolvconf >/dev/null 2>&1; then + if ! resolvconf -u; then + log "resolvconf -u failed; $RESOLV_CONF was not regenerated" + exit 1 + fi + fi fi -# systemd-resolved owns its stub file and rewrites anything appended to -# it. The equivalent setting there is a resolved.conf drop-in, which is -# a different change than this script makes -- say so rather than -# writing to a file that will be overwritten. +# systemd-resolved owns its stub file and rewrites anything appended to it, +# and `single-request` is a glibc resolv.conf option with no resolved.conf +# equivalent -- so there is nothing this script can do here. Exit non-zero: +# the unit would otherwise record success while the workaround is inactive, +# which is the failure mode this whole script exists to avoid. if [ -L "$RESOLV_CONF" ] && readlink -f "$RESOLV_CONF" | grep -q "systemd"; then - log "$RESOLV_CONF is managed by systemd-resolved; not modifying it." - log "If lookups are slow, see 'man resolved.conf' -- the resolver options" - log "there are set per-link, not through resolv.conf." - exit 0 + log "$RESOLV_CONF is managed by systemd-resolved." + log "'options single-request' is a glibc resolv.conf option and has no" + log "resolved.conf equivalent, so it cannot be applied on this host." + log "If external API calls are slow, the workaround is to stop using the" + log "systemd-resolved stub (see 'man systemd-resolved', NSS/resolv.conf modes)." + exit 1 fi if already_applied "$RESOLV_CONF"; then @@ -60,10 +71,24 @@ if already_applied "$RESOLV_CONF"; then exit 0 fi -if [ -w "$RESOLV_CONF" ] || [ ! -e "$RESOLV_CONF" ]; then - echo "$OPTION" >> "$RESOLV_CONF" - log "added to $RESOLV_CONF" -else +if [ ! -w "$RESOLV_CONF" ] && [ -e "$RESOLV_CONF" ]; then log "cannot write $RESOLV_CONF (run with sudo?)" exit 1 fi + +# A NetworkManager-generated resolv.conf is regenerated on every connection +# change, not only at boot -- and this unit is oneshot with RemainAfterExit, +# so it will not re-run within the same boot to put the option back. Say so +# rather than implying the fix is permanent. Nothing is silently swallowed: +# the append below still happens and still works until the next renewal. +if grep -qs "Generated by NetworkManager" "$RESOLV_CONF" \ + && [ ! -d "$(dirname "$RESOLVCONF_TAIL")" ]; then + log "NOTE: $RESOLV_CONF is generated by NetworkManager and has no" + log "resolvconf tail directory to write to. The option is being added, but" + log "NetworkManager will drop it on the next connection renewal, and this" + log "unit does not run again until the next boot. If lookups go slow again" + log "before a reboot, re-run this script." +fi + +echo "$OPTION" >> "$RESOLV_CONF" +log "added to $RESOLV_CONF" diff --git a/scripts/utils/pixlet_config_editor.sh b/scripts/utils/pixlet_config_editor.sh index 7c57de47b..8437623bc 100644 --- a/scripts/utils/pixlet_config_editor.sh +++ b/scripts/utils/pixlet_config_editor.sh @@ -15,11 +15,14 @@ # actually editing. # # Usage: -# ./scripts/utils/pixlet_config_editor.sh # list installed apps -# ./scripts/utils/pixlet_config_editor.sh # edit, on localhost -# ./scripts/utils/pixlet_config_editor.sh --lan # reachable from the LAN +# ./scripts/utils/pixlet_config_editor.sh # list installed apps +# ./scripts/utils/pixlet_config_editor.sh # edit +# +# Binds loopback only, and there is deliberately no flag to change that: +# `pixlet serve` has no authentication, and anything that can reach it can +# rewrite the app's config. To edit from another machine, forward the port -- +# which authenticates as SSH and leaves nothing listening on the LAN: # -# On localhost, reach it from another machine over SSH instead of --lan: # ssh -L 8080:localhost:8080 pi@ledpi.local set -eu @@ -27,12 +30,10 @@ set -eu PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) APPS_DIR="$PROJECT_ROOT_DIR/starlark-apps" PORT="${PIXLET_EDITOR_PORT:-8080}" +# Loopback only. See the header: pixlet serve is unauthenticated. BIND_HOST="127.0.0.1" APP_ID="${1:-}" -if [ "${2:-}" = "--lan" ]; then - BIND_HOST="0.0.0.0" -fi list_apps() { if [ -d "$APPS_DIR" ]; then @@ -41,7 +42,7 @@ list_apps() { } if [ -z "$APP_ID" ]; then - echo "Usage: $0 [--lan]" + echo "Usage: $0 " echo "" echo "Installed apps:" list_apps || true @@ -124,17 +125,11 @@ fi echo "" echo "Editing: $APP_ID" echo "App file: $STAR_FILE" -if [ "$BIND_HOST" = "0.0.0.0" ]; then - echo "URL: http://$(hostname):$PORT/" - echo "" - echo "⚠ Listening on all interfaces with no authentication. Anyone on this" - echo " network can change this app's config while the session is open." -else - echo "URL: http://localhost:$PORT/" - echo "" - echo "Listening on localhost only. From another machine, forward the port:" - echo " ssh -L $PORT:localhost:$PORT $(whoami)@$(hostname)" -fi +echo "URL: http://localhost:$PORT/" +echo "" +echo "Listening on localhost only -- pixlet serve has no authentication." +echo "From another machine, forward the port:" +echo " ssh -L $PORT:localhost:$PORT $(whoami)@$(hostname)" echo "" echo "Changes save straight to the real config as you make them." echo "Press Ctrl+C when finished - the display restarts automatically." diff --git a/test/test_api_v3_display_modes.py b/test/test_api_v3_display_modes.py index 2977f6896..d77546fab 100644 --- a/test/test_api_v3_display_modes.py +++ b/test/test_api_v3_display_modes.py @@ -6,6 +6,7 @@ manifest.json off disk and reimplemented PluginManager's own fallbacks. """ +import json from unittest.mock import MagicMock import pytest @@ -102,3 +103,55 @@ def test_a_plugin_with_no_declared_modes_still_appears(self, client, api_v3_modu modes = _modes(client.get('/api/v3/display/modes')) assert modes['starlark-apps']['plugin_id'] == 'starlark-apps' + + +class TestOneBadConfigSectionDoesNotBlankTheList: + """config.json can hold a non-dict under a plugin id. + + DisplayController guards the same shape, so it happens in practice. Here it + used to raise AttributeError mid-loop and answer 500 with no modes at all -- + and the MQTT bridge builds every one of its entities from this list, so one + hand-edited section would empty the Home Assistant dropdown. + """ + + @pytest.fixture + def client_with_bad_section(self, api_v3_module, api_v3_client): + pm = api_v3_module.api_v3.plugin_manager + pm.plugin_manifests = MANIFESTS + pm.discover_plugins = MagicMock(return_value=list(MANIFESTS)) + pm.get_plugin_display_modes = MagicMock( + side_effect=lambda pid: MANIFESTS[pid]['display_modes']) + api_v3_module.api_v3.config_manager.load_config = MagicMock(return_value={ + 'clock-simple': {'enabled': True}, + 'football-scoreboard': "true", # a string, not an object + 'ledmatrix-weather': {'enabled': True}, + }) + return api_v3_client + + def test_the_endpoint_still_answers(self, client_with_bad_section): + assert client_with_bad_section.get('/api/v3/display/modes').status_code == 200 + + def test_the_healthy_plugins_are_still_listed(self, client_with_bad_section): + modes = _modes(client_with_bad_section.get('/api/v3/display/modes')) + assert 'clock-simple' in modes and 'weather' in modes + + def test_the_bad_section_is_treated_as_disabled(self, client_with_bad_section): + modes = _modes(client_with_bad_section.get('/api/v3/display/modes')) + assert 'nfl_live' not in modes + + def test_a_failure_is_reported_the_way_every_other_handler_reports_one( + self, api_v3_module, api_v3_client): + """describe_exception, per test_web_error_detail's contract -- an + opaque "see logs for details" is what that test exists to prevent.""" + api_v3_module.api_v3.plugin_manager.discover_plugins = MagicMock( + side_effect=RuntimeError("disk is gone")) + resp = api_v3_client.get('/api/v3/display/modes') + assert resp.status_code == 500 + assert 'disk is gone' in resp.get_json()['details'] + + def test_credentials_in_the_exception_are_redacted(self, api_v3_module, api_v3_client): + """describe_exception is what makes returning detail safe.""" + api_v3_module.api_v3.plugin_manager.discover_plugins = MagicMock( + side_effect=RuntimeError("GET https://x/y?api_key=SEC123 failed")) + body = api_v3_client.get('/api/v3/display/modes').get_json() + assert 'SEC123' not in json.dumps(body) diff --git a/test/test_mqtt_bridge.py b/test/test_mqtt_bridge.py index 9840e34be..7a3c75010 100644 --- a/test/test_mqtt_bridge.py +++ b/test/test_mqtt_bridge.py @@ -9,9 +9,10 @@ import importlib.util import json +import logging import sys from pathlib import Path -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch import pytest @@ -264,3 +265,65 @@ def test_a_non_numeric_port_is_a_clear_error(self, bridge_module, tmp_path): path.write_text(json.dumps({"mqtt_port": "not-a-port"})) with pytest.raises(bridge_module.ConfigError): bridge_module.load_config(str(path)) + + +class TestTheExampleConfigIsSecureByDefault: + """The installer copies bridge_config.example.json verbatim on first run. + + Without TLS the broker password and every display command cross the network + in cleartext, so the shipped default has to be the safe one -- a plaintext + broker is a deliberate edit, not something you get by not reading. + """ + + @pytest.fixture + def example(self): + path = (Path(__file__).resolve().parent.parent + / "integrations" / "mqtt_bridge" / "bridge_config.example.json") + if not path.exists(): + pytest.skip("mqtt bridge example config is not present") + return json.loads(path.read_text(encoding="utf-8")) + + def test_tls_is_on(self, example): + assert example["mqtt_tls"] is True + + def test_the_port_is_the_tls_one(self, example): + """1883 with mqtt_tls on would just fail to connect.""" + assert example["mqtt_port"] == 8883 + + def test_certificate_verification_is_not_disabled(self, example): + assert example.get("mqtt_tls_insecure", False) is False + + def test_no_password_ships_in_the_example(self, example): + assert example["mqtt_password"] is None + + def test_the_example_loads(self, bridge_module, tmp_path): + """It is copied verbatim, so it must survive load_config.""" + path = (Path(__file__).resolve().parent.parent + / "integrations" / "mqtt_bridge" / "bridge_config.example.json") + config = bridge_module.load_config(str(path)) + assert config["mqtt_tls"] is True and config["mqtt_port"] == 8883 + + +class TestCleartextIsCalledOut: + """Turning TLS off is allowed -- the Mosquitto add-on is plaintext on 1883 -- + but it should not be silent when a password is going over it.""" + + def test_a_password_without_tls_warns(self, bridge_module, caplog): + with caplog.at_level(logging.WARNING): + warned = bridge_module.warn_if_cleartext( + {"mqtt_tls": False, "mqtt_password": "hunter2"}) + assert warned is True + assert "unencrypted" in caplog.text + + def test_the_warning_does_not_repeat_the_password(self, bridge_module, caplog): + with caplog.at_level(logging.WARNING): + bridge_module.warn_if_cleartext({"mqtt_tls": False, "mqtt_password": "hunter2"}) + assert "hunter2" not in caplog.text + + def test_no_password_means_nothing_to_lose(self, bridge_module): + assert bridge_module.warn_if_cleartext( + {"mqtt_tls": False, "mqtt_password": None}) is False + + def test_tls_on_does_not_warn(self, bridge_module): + assert bridge_module.warn_if_cleartext( + {"mqtt_tls": True, "mqtt_password": "hunter2"}) is False diff --git a/web_interface/blueprints/api_v3.py b/web_interface/blueprints/api_v3.py index da02779f9..a9079b051 100644 --- a/web_interface/blueprints/api_v3.py +++ b/web_interface/blueprints/api_v3.py @@ -2552,7 +2552,20 @@ def get_display_modes(): modes = [] for plugin_id, manifest in sorted(api_v3.plugin_manager.plugin_manifests.items()): - enabled = bool(full_config.get(plugin_id, {}).get('enabled', False)) + # A hand-edited or migrated config.json can hold a non-dict under a + # plugin id; DisplayController._reconcile guards the same shape, so + # it happens in practice. Without this, .get() raises AttributeError, + # the loop aborts and the endpoint answers 500 with no modes at all + # -- one bad section would blank every entity the MQTT bridge builds + # from this list. + plugin_config = full_config.get(plugin_id) + if not isinstance(plugin_config, dict): + if plugin_config is not None: + logger.warning( + "Config for plugin %r is %s, not an object; treating it as disabled", + plugin_id, type(plugin_config).__name__) + plugin_config = {} + enabled = bool(plugin_config.get('enabled', False)) if not enabled and not include_disabled: continue plugin_name = (manifest or {}).get('name') or plugin_id @@ -2571,6 +2584,12 @@ def get_display_modes(): return jsonify({'status': 'success', 'data': {'modes': modes}}) except Exception as exc: + # describe_exception, not a bare message: test_web_error_detail.py + # enforces that every handler here returns it, because a device whose + # storage is failing otherwise answers "see logs for details" from the + # log viewer too. It redacts credentials out of the exception text. + # CodeQL flags this as stack-trace exposure across all ~75 handlers; + # it is the project's deliberate, reviewed trade-off. logger.error('Error in get_display_modes', exc_info=True) return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details', 'details': describe_exception(exc)}), 500