From ff21321f1101c737baa73f9856ef2ed5ce0e5257 Mon Sep 17 00:00:00 2001 From: Buba Date: Mon, 18 May 2026 23:02:09 +0200 Subject: [PATCH] Add debugging skills and repo-specific patterns for container services --- .../skills/debug-container-service/SKILL.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 .claude/skills/debug-container-service/SKILL.md diff --git a/.claude/skills/debug-container-service/SKILL.md b/.claude/skills/debug-container-service/SKILL.md new file mode 100644 index 00000000..c8d95fc3 --- /dev/null +++ b/.claude/skills/debug-container-service/SKILL.md @@ -0,0 +1,183 @@ +--- +name: debug-container-service +description: > + Debug, troubleshoot, and fix container-based services (Docker, Podman) configured in this + homelab infrastructure repo. Use when a service is broken, crashing, returning errors, or + behaving unexpectedly. Triggers on: "X is not working", "X gives an error", "debug service X", + "fix container X", "troubleshoot X", service crashes, 500/400 errors from a running service, + unexpected behavior after an update, config changes that broke something. Also use when asked + to investigate why a service stopped or started failing. +--- + +# Container Service Debugging + +Work through the steps below in order. **Stop as soon as the problem is resolved** — do not continue to further steps unnecessarily. + +## Before You Begin + +Make sure you have: +- The error message or symptom (ask the user if not provided) +- The service name in `category/service-name` format (e.g. `ai/open-webui`, `security/traefik`, `media/video/jellyfin`) + +The compose file is at `docker///.yaml`. If the user mentions an error but not the service name, ask. If the service is obvious from context, proceed. + +--- + +## Step 1 — Get Compose Config and Identify Services + +Use the MCP tool to read the resolved compose configuration: + +``` +mcp__infra-mcp__container-service-config(service_name="category/service-name") +``` + +From the config, identify: +- **Container name** (needed for `docker logs` — usually set explicitly as `container_name:`) +- **Volumes** (paths to check for file-based logs) +- **Environment variables** (may reveal misconfiguration) +- **Image tag** (needed for source code lookups in Steps 4–7) + +Environment variables are loaded from `.env` files in this precedence order (later overrides earlier): +1. `config/docker/.env` — common to all hosts +2. `config/docker//.env` — host-specific +3. `config/docker/.env.` — service-specific (all hosts) +4. `config/docker//.env.` — host + service specific + +**Check indirect dependencies too.** Many errors originate upstream of the failing service: +- **Traefik** — reverse proxy; routing or TLS misconfiguration shows up as connection errors in the app +- **Authelia** — a 401/403 may be Authelia rejecting the session, not the app itself +- **Cloudflared** — tunnel connectivity issues affect all externally-exposed services +- **Ollama / LiteLLM** — if an AI frontend errors, check the model provider first + +--- + +## Step 2 — Check Docker Logs (Always Do This First) + +**Never make config changes before reading the actual logs.** The log output reveals: +- The exact error message and exception type +- The file and line number where the error occurs — this identifies which code path is failing, which determines which fix applies +- Whether the problem is in this service or in a dependency it's calling + +```bash +docker logs --tail 100 2>&1 +docker logs --tail 200 2>&1 | grep -i "error\|exception\|warn\|fail\|traceback" +``` + +Or use the labctl wrapper (handles env var substitution): +```bash +scripts/labctl.py service logs category/service-name --tail 200 +scripts/labctl.py service logs category/service-name --follow +scripts/labctl.py service logs category/service-name --since 30m +scripts/labctl.py service logs category/service-name --timestamps +``` + +**Key insight from logs:** Pay attention to the source location (e.g. `module.file:function:line`). An error in `process_chat` vs `images_endpoint` are different code paths that may require completely different fixes, even if the symptom looks the same. + +If dependent services are suspected, check their logs too. + +--- + +## Step 3 — Check File-Based Logs + +Some services write additional structured logs to their data volume. Check: + +1. Get the host-side volume paths for the container: + ```bash + docker inspect --format '{{range .Mounts}}{{.Source}}{{"\n"}}{{end}}' + ``` +2. Look for log files in those paths: + ```bash + find -name "*.log" -type f | head -20 + find -path "*/logs/*" -type f | head -20 + ``` +3. Read recent entries of any log files found. + +--- + +## Step 4 — Search Documentation + +The GitHub repository URL is usually in a comment at the top of the compose file. Use it to find the `/` for the commands below. + +Use Context7 to search the service's official documentation for the specific error or feature: + +``` +mcp__context7__resolve-library-id(libraryName="", query="") +mcp__context7__query-docs(libraryId="...", query="") +``` + +**Critical rule:** Before adding any environment variable or config option to a service YAML, **verify the exact name exists in the source code or official docs**. Do not guess or derive variable names from patterns — look them up. + +Use the GitHub API to read the actual source for the version in use: +```bash +gh api repos///contents/?ref= --jq '.content' | base64 -d | grep -i "" +``` + +--- + +## Step 5 — Search GitHub Issues + +If the error message is specific (a stack trace fragment, a quoted error string, an unusual behavior), search the project's issue tracker early. Many errors in popular open-source services are already documented with working fixes. + +```bash +gh issue list -R / --search "" --state all --limit 10 +``` + +Read the comments on relevant issues — the fix is often in the comments, not the issue body. Look for: +- Confirmed fixes (env vars, config changes) +- Workarounds +- Which version introduced or fixed the problem + +```bash +gh issue view -R / --comments +``` + +--- + +## Step 6 — Check Version / Release Notes + +If the issue appeared after an update, or if a fix exists in a newer version: + +```bash +# List available image tags +mcp__infra-mcp__list-container-tags(image="") + +# Browse recent release notes +gh api repos///releases --jq '.[0:5] | .[] | {tag: .tag_name, body: .body[:400]}' +``` + +**Upgrading:** If a newer version contains the fix, update the image tag in the service YAML file (`docker/category/service/service.yaml`) and recreate the container. + +**Downgrading:** If a recent update introduced the regression, note the last known-good version and offer to roll back. + +--- + +## Step 7 — Source Code Inspection and Patch (Last Resort) + +Only reach this step after Steps 1–6 have not resolved the issue. + +**Before making any change:** Explain to the user exactly what you intend to change in the source, why it fixes the issue, and what the trade-offs are. Ask for confirmation. + +Read the relevant source for the exact deployed version: +```bash +gh api repos///contents/?ref= --jq '.content' | base64 -d + +# List directory contents +gh api repos///contents/?ref= --jq '.[].path' +``` + +Choose the least invasive fix, in this order: + +1. **Environment variable or config change** — no files to mount, most maintainable +2. **Mount a patched file via Docker volume** — add a bind mount in the service YAML pointing to a patched copy stored in the repo (e.g. `./patched_file.py:/app/path/file.py:ro`); document clearly why it exists +3. **Custom Docker image** — only if a volume mount is not feasible; significantly increases maintenance burden + +**Always revert wrong fixes before applying the correct one.** Accumulating incorrect changes obscures the actual state and makes future debugging harder. + +After applying any fix, recreate the container to pick up changes: +```bash +scripts/labctl.py service recreate category/service-name +# or +mcp__infra-mcp__container-service-recreate(service_name="category/service-name") +``` + +Then re-check the logs (Step 2) to confirm the fix worked.