Importer perf: pre-fetch Folder pk map + narrow prune prefetch - #630
Merged
Merged
Conversation
Previously _prepare_moved_comic fired two SELECTs per moved comic:
Folder.objects.get(path=new_path.parent) for the parent_folder FK,
then Folder.objects.filter(path__in=new_path.parents) for the
folders M2M rebuild. For a 600k-comic library reorg that was
~1.2M small SELECTs.
_bulk_comics_moved_ensure_folders already runs first and creates
any missing destination folders, so by the time we hit the prepare
loop the table contains every parent path the loop can possibly
need. One values_list pass at the top of _bulk_comics_move_prepare
builds a {path: pk} map covering this library; the per-comic loop
then does dict lookups instead of SQL.
Bonus correctness fix: the prebuilt map is filtered by
library=self.library, while the prior Folder.objects.get(path=...)
was not library-scoped — so on a multi-library install a
parent_folder could resolve to another library sharing the same
on-disk path. Bracket access on the parent lookup raises KeyError
on a genuinely missing parent, which the broad except below catches
identically to the prior DoesNotExist.
Implements hot path A from tasks/importer-perf/03-query-prune.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously query_prune_comic_m2m_links called prefetch_related with the full COMIC_M2M_FIELD_NAMES (14 M2Ms) per batch, and query_prune_comic_fk_links called select_related with the full COMIC_FK_FIELD_NAMES (12 FKs). Most imports only touch a handful of each — credits, tags, characters on the M2M side; volume, parent_folder, age_rating on the FK side — so the wide prefetch fired 10+ wasted IN-queries and hauled their through-rows into Python memory only to be ignored. Compute the actually-referenced field set once per phase by walking LINK_M2MS / LINK_FKS keys, then pass that subset through to the batch helpers. For a typical import this drops the prune phase from 14 prefetch IN-queries to 3-5, and from a 12-table join to a 3-table join. Also hoist self.status_controller.update(status) out of the per-(through-row, field) inner loops in _query_prune_comic_m2m_links_field and _query_prune_comic_fk_links_comic. The controller already rate-limits at _UPDATE_DELTA so the inner calls only burned CPU on the early-return path; one update per (comic, field) and one per comic respectively keeps the visible refresh cadence identical. The unused module-level _QUERY_LINK_FK_PRUNE_ONLY constant is dropped; only_fields is now computed inline from the narrowed select_related set. Implements hot paths B and C from tasks/importer-perf/03-query-prune.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced Apr 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements
tasks/importer-perf/03-query-prune.mdfrom PR #627. Third surgical batch after #628 (link prepare) and #629 (create FKs).Two independently revertable commits:
f2bddf25— moved comics: pre-fetch Folder pk map (headline)_prepare_moved_comicpreviously fired two SELECTs per moved comic (parent folder + ancestor M2M rebuild). For a 600k-comic library reorg that was ~1.2M small SELECTs._bulk_comics_moved_ensure_foldersalready runs first and creates any missing destination folders, so a singlevalues_listpass at the top of the prepare loop builds a{path: pk}map covering this library; the per-comic loop becomes dict lookups.Bonus correctness fix: the prebuilt map is
library=self.library-scoped, while the priorFolder.objects.get(path=...)was not — on multi-library installs aparent_foldercould resolve to another library sharing the same on-disk path. Tightens the search to this library.9b3af11f— prune: narrow prefetch/select_related, hoist status updatequery_prune_comic_m2m_linkswas callingprefetch_related(*COMIC_M2M_FIELD_NAMES)(14 M2Ms) per batch even though most imports only touch credits/tags/characters. Now computes the LINK_M2MS-referenced subset once per phase and prefetches only that — typically 3-5 IN-queries instead of 14, with proportional drop in through-row memory pressure.query_prune_comic_fk_linkssimilarly narrowed from a 12-tableselect_relatedJOIN to a 3-table JOIN._query_prune_comic_m2m_links_fieldand_query_prune_comic_fk_links_comicwere callingstatus_controller.update()inside their per-through-row / per-FK-field inner loops. The controller already rate-limits at_UPDATE_DELTA = 5s, so the inner calls only burned CPU on the early-return path. Hoisted to once per (comic, field) and once per comic respectively; visible refresh cadence is identical._QUERY_LINK_FK_PRUNE_ONLYconstant.Query count summary
Test plan
make fixcleanmake lint-pythonclean (0 errors, 0 warnings)pytest tests/importer/ tests/test_search_fts.py— 7 passedComic.parent_folderandComic.foldersM2M rows match the pre-PR baseline🤖 Generated with Claude Code