Cover daemon perf: plan

[ajslater]

Summary

Plan for codex/librarian/covers/. Single-file plan because the
surface is small (6 files, ~440 LOC) — no meta split needed.

Pure planning deliverable, no code changes.

Eleven ranked findings

#	Finding	Severity
F1	Pre-fetch DB paths in the parent (N serial SELECTs → 1)	refactor
F2	Skip-existing pre-filter in the parent	small win
F3	ProcessPoolExecutor for the image pipeline	HIGH
F4	Lazy + persistent pool lifecycle	bundles with F3
F5	CODEX_COVER_WORKERS setting	bundles with F3
F6	Single-pk inline fast path	speculative — defer
F7	purge_cover_paths parallelization	skip
F8	Stale create_cover_from_path docstring	cleanup
F9	Empty @dataclass decorators on cover tasks	cosmetic
F10	Status update batching	speculative — defer
F11	Memory ceiling guidance	documentation

Headline finding

F3 — the cover thread today serializes a CPU-bound image
pipeline through a single worker:

for pk in pks:
    self._bulk_create_comic_cover(pk, status, custom=custom)

Each iteration runs comicbox.get_cover_page + PIL
Image.thumbnail(LANCZOS, reducing_gap=3.0) + save("WEBP", method=6). On a multi-core box, 7 cores sit idle.

Plan recommends concurrent.futures.ProcessPoolExecutor (not
threads — PIL releases the GIL inconsistently across encoder
paths). Worker function is top-level (picklable), accepts
pre-resolved (pk, db_path, cover_path_str, custom) tuples,
returns (pk, thumb_bytes, error_msg). Parent uses
as_completed to stream results, writes to disk via the
existing atomic-replace save_cover_to_cache, updates status
per completion.

F1 (pre-fetch DB paths) and F2 (skip-existing pre-filter) are
called out separately because they need to land first — F3's
worker subprocess shouldn't carry a Django connection, and the
batched path fetch + skip-existing filter remove that
dependency.

Suggested ordering

1. F1 + F2 — pure refactor, no thread changes. Land first;
   F3 builds on it.
2. F8 + F9 — cleanups. Bundle anywhere.
3. F3 + F4 + F5 — the big change. Process pool, lazy
   creation, settings knob. Single PR; the unit of risk is the
   subprocess introduction.

Risks flagged

• comicbox + PIL subprocess safety (expected fine, verify via
  smoke test against a populated dev DB).
• COMICBOX_CONFIG import cost in workers — pass the dict as a
  worker arg vs. re-importing Django settings per subprocess.
• Loguru cross-process logging — workers return error strings
  for the parent to log, sidesteps the issue.
• Pool shutdown on hard kill — ProcessPoolExecutor atexit
  handler should cover it; signal handler if not.
• RAM ceiling: ~100 MB/worker × 8 workers ≈ 800 MB peak.
  Documented; honor CODEX_COVER_WORKERS for memory-tight
  installs.

References

• Plan: tasks/coverd-perf/00-plan.md
• codex/librarian/scribe/importer/create/comics.py:173 — main
  CoverCreateTask producer (post-import batches)
• codex/views/browser/cover.py:104 — the 202-poll
  per-request producer (single pk per task)

🤖 Generated with Claude Code