Search indexer perf: plan

[ajslater]

Summary

Plan for codex/librarian/scribe/search/. Single-file plan
because the surface is small (8 files, ~750 LOC).

Pure planning deliverable, no code changes.

Eight findings, three are tier-1

#	Finding	Severity
F1	_prefetch_related_fts_query is dead code	refactor
F2	3 column-name mismatches → silent data drop	correctness + perf
F3	Megaquery (10 GROUP_CONCAT aggregates + LEFT JOINs) materializes cartesian product	headline perf
F4	ComicFTS.exists() per loop iteration	small win
F5	time() → monotonic() for elapsed reporting	low
F6	clear_search_index() slow on FTS5 due to per-row token removal	defer
F7	M2M .add() / .remove() doesn't bump Comic.updated_at	out of scope
F8	Raw-SQL INSERT INTO codex_comicfts SELECT ...	defer

Standout findings

F1 — prefetch_related + .values() dead code

Empirically verified — when a queryset chains
prefetch_related(...) with .values(), prefetches don't fire:

>>> qs = Comic.objects.prefetch_related("characters", "credits", "genres").values("pk")
>>> with CaptureQueriesContext(connection) as ctx:
...     list(qs)
>>> len(ctx.captured_queries)
1   # only the main SELECT — zero prefetch queries

Django's prefetch_related_objects walks the result cache
looking for FK descriptors, but .values() rows are plain
dicts with no descriptors. The 11 prefetch declarations in
the indexer's hot path contribute zero query work — pure
documentation. The associated comment about "1000 sqlite query
depth" is irrelevant since nothing's being prefetched.

F2 — Column-name mismatches → silent data drop

prepare.py:_COMIC_KEYS lists what the consumer reads:

_COMIC_KEYS = (..., "fts_credits", "fts_sources", "fts_story_arcs", ...)

sync.py:_M2M_FTS_RELS produces annotations under different
aliases:

_M2M_FTS_RELS = (..., "credits__person", "identifiers__source",
                      "story_arc_numbers__story_arc", ...)
# annotation alias becomes f"fts_{rel}" — i.e. fts_credits__person, etc.

The DB pays for three GROUP_CONCAT aggregations:
fts_credits__person, fts_identifiers__source,
fts_story_arc_numbers__story_arc. The Python consumer reads
fts_credits, fts_sources, fts_story_arcs. Mismatch — the
consumer's comic.get(key, "") filter silently drops them.

Two failure modes ride this single bug:

• Perf waste: three GROUP_CONCATs run as part of the
  megaquery's cartesian product cost, results discarded.
• Correctness regression: credits / sources /
  story_arcs columns of codex_comicfts are empty for any
  sync-built entry. Searching for a credit-person name,
  identifier source, or story-arc title silently returns no
  results. (prepare_import_fts_entry populates these
  correctly via comicbox payload keys, so import-built entries
  work — only the sync path is broken.)

Likely a regression: M2M rels were renamed to include the FK
suffix (credits → credits__person so GROUP_CONCAT walks the
right column), but the Python consumer keys weren't updated.

Fix: rename _M2M_FTS_RELS to short-name keys and use a
separate map for the GROUP_CONCAT source field. Same SQL gets
generated; the alias matches _COMIC_KEYS; the data flows
through.

F3 — Megaquery → per-M2M UNION batching

The headline perf finding. The current shape is one SELECT with:

• 10 LEFT JOINs to FK tables
• 10 LEFT JOINs to M2M through tables × 2 = 20 more
• 10 GROUP_CONCAT(... DISTINCT ORDER BY ...) aggregations
• GROUP BY comic.id to collapse the cartesian product

For a comic with 5 characters × 3 credit-people × 2 genres × 4
tags, the temp table before GROUP BY has 5 × 3 × 2 × 4 × ...
rows. SQLite materializes this, sorts for DISTINCT in each
GROUP_CONCAT, then GROUP BYs to collapse. Memory + I/O blow up
on richly-tagged libraries.

Apply the OPDS metadata batching pattern
(codex/views/opds/metadata.py:get_m2m_objects_by_comic):

1. One query for the comic + simple FK columns
   (select_related for the 10 FK joins; no M2M).
2. Ten independent queries for each M2M, each shaped
   GROUP_CONCAT(<m2m>__name) GROUP BY <m2m>__comic_id.
3. Stitch the per-pk dicts in Python.

11 queries per batch instead of 1 megaquery. Sounds worse, but
each individual query walks one M2M's index — no cartesian
product. Cost goes from O(product of M2M counts × batch size)
to O(sum of M2M-table-rows per relation).

Suggested ordering

1. F1 — pure cleanup. Land first, smallest diff.
2. F2 — correctness + perf. Behavior change (sync entries
   now have credits/sources/story_arcs populated) needs a test
   pass on a populated dev DB before merge.
3. F4 + F5 — small bundle (exists() hoist + monotonic
   clock).
4. F3 — the headline. Largest diff, biggest payoff. Land
   after the smaller ones so a regression bisect points at the
   right commit.
5. F6 / F7 / F8 — deferred. Open follow-up issues.

Risks flagged

• F2 wire change: searches for credit / source / story-arc
  queries shift from "always empty" to "populated". Could be
  perceived as a regression by anyone relying on the empty-
  results behavior. Roll out with a release note.
• F3 query reshape: if per-M2M UNION batching trips a
  SQLite planner edge case, perf could regress on small
  libraries (where the megaquery is small enough not to blow
  up). Microbench against a range of library sizes
  (1k / 10k / 100k comics).
• F6 deferred: FTS5 token re-indexing during DELETE is
  fundamental; bulk INSERT doesn't help with token cost.
  Don't promise wins beyond what F1+F2+F3 deliver for sync.

References

• codex/librarian/scribe/search/sync.py — the hot loop
• codex/librarian/scribe/search/prepare.py — the consumer
  with the name mismatches
• codex/views/opds/metadata.py:get_m2m_objects_by_comic —
  the per-M2M UNION batching pattern F3 mirrors
• PR Serializers perf 03: OPDS v2 batching + link/unused cleanup #615 (OPDS v2 batching) — same shape applied to OPDS
• PR Threads + worker: gate type-only imports + monotonic clock #623 / Status controller perf: monotonic clock + drop wasted SELECT + gate type-only imports #624 / Search indexer perf: plan #625 — the time() → monotonic() fix
  applied to other librarian modules

🤖 Generated with Claude Code