perf(api): make gzip compression level configurable - #9441
Merged
Merged
Conversation
The gallery list/name routes and the auth dependencies were declared `async def` while calling synchronous SQLite services, so their database work ran on the event loop. For its whole duration the process served no other request and delivered no socket.io event, which users experienced as the backend freezing mid-generation rather than as a slow gallery. Declaring them `def` hands them to Starlette's threadpool instead. Measured against a 200k-image, 1.7 GB database: the latency of an unrelated request issued while a gallery name query is in flight drops from 1662 ms to 4 ms (search) and from 2298 ms to 881 ms (no search). The queries themselves are unchanged; only the loop is freed. The residual 881 ms in the no-search case is response serialization of 202k items, which is tracked separately. Adds a regression test that stubs a blocking service call and asserts an unrelated route still answers during it, plus a contributor doc describing the rule.
…y ones The name list that drives the virtualized gallery wrapped every entry in an object carrying a `kind` discriminator. Building those models cost 820ms of the 2225ms service call on a 200k-item library, and every consumer threw the field away — `itemRefsToNames` mapped it off immediately and each caller re-derived the kind from the file extension via `isVideoName`. Adds `GET /v1/gallery/item_names`, returning a flat name list in the same shape as the image-only `ImageNamesResult`. An optional `created_date` filter subsumes the separate by-date virtual-board route, so regular boards and virtual dates now share one endpoint, one cache and one query-args selector instead of a skipToken branch duplicated across the grid hook, range selection and both auto-select listeners. Measured on a 200k-image, 1.7 GB database: 2.51s -> 1.57s per request, 8.48 MB -> 3.85 MB of response, and the residual event-loop stall from serializing the response drops from 466ms to 102ms at p95. Existing integrations still call the old routes, so all five legacy name endpoints keep working and are marked `deprecated=True` with a pointer to the replacement.
Starlette's GZipMiddleware compresses every response type except text/event-stream, so every image and video the gallery serves was being deflate-compressed a second time. Measured: a 1024x1024 PNG (3.00 MB) costs 52ms of event-loop time to gzip and comes back at 3.01 MB — larger than it went in; a 2048x2048 PNG costs 210ms for the same non-result. Compression runs on the event loop, so that time is a full stall of the process. With auto-switch enabled the UI fetches the full image after every generated image, so the cost lands repeatedly during a batch. Replaces it with a content-type-aware subclass that compresses an allowlist of text, JSON, XML and SVG responses and passes everything else through. The UI bundle and the API's JSON keep their compression unchanged. Lowering compresslevel is not an alternative for this case: on already-compressed input level 1 costs 51ms against level 9's 52ms, because deflate still scans the whole body. Making the level configurable is worthwhile for the *compressible* path and is tracked separately. Note for deployments: media responses no longer carry Content-Encoding: gzip.
The pin sat at 0.118.3 with a comment guessing the OpenAPI crash on 0.119 was "probably Invoke's [bug], because we are doing something unusual with AnyInvocation". It was not: fastapi/_compat/v2.py assumed every field mapping carries a `$ref` and raised KeyError otherwise. Upstream fixed it in 0.124.0 with no change needed here. Two later changes needed adapting to, both of which fail silently: - 0.130 emits `contentMediaType: application/octet-stream` instead of `format: binary` for file uploads. typegen.js mapped only the latter to `Blob`, so upload call sites would have started typing their `File` argument as `string`. It now maps both. - 0.141 keeps an included router as a single node in `app.routes` instead of copying its routes into it. The default-deny auth guard walked `app.routes` looking for APIRoute instances and found 2 of 197 — passing while inspecting almost nothing. It now walks `iter_route_contexts`, the traversal FastAPI's own OpenAPI generation uses, and asserts a floor on the route count so going blind fails loudly instead. Schema changes are limited to ValidationError gaining the optional `input`/`ctx` fields; upload fields still resolve to Blob. Starlette stays at 0.48.0.
Package A converted the eight gallery and search routes that caused the reported multi-minute stalls. The same defect was present across the rest of the API: 167 route handlers were declared `async def` while awaiting nothing, so their synchronous service calls ran on the event loop. Each one stalls the entire process for its duration - no other request served, no socket.io event delivered - which is why the symptom looked like the application freezing rather than one slow endpoint. Candidates were identified by AST rather than by hand: `async def` route handlers with no `await`, `async with` or `async for` anywhere in the body, cross-checked for references to asyncio, anyio or the loop. Two flagged candidates were false positives (both the word "loop" in a comment). The diff is 167 signature lines plus one signature that ruff collapsed onto a single line once `async ` was removed. Adds tests/app/routers/test_no_blocking_async_routes.py, which enforces the rule for every handler including ones written later - a per-route test cannot cover a route that does not exist yet, and this failure mode is invisible until a user has a large enough library to notice. Two tests that invoked route handlers directly were updated to call them as the plain functions they now are.
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant,
dunkeroni and
lstein
as code owners
August 2, 2026 03:21
Starlette's GZipMiddleware compresses at level 9, the slowest setting, and there was no way to change that. Compression runs on the event loop, so it stalls every other request and every socket.io event for its duration. Measured on the flat image-name list of a 200k-image library (8.48 MB of JSON): level 1 takes 16.4ms and returns 6.1% of the input, level 9 takes 90.2ms and returns 5.7%. Level 9 spends 5.5x the event-loop time to save 0.4 percentage points of bandwidth — a poor deal for a locally-served app, where the saved bandwidth is worthless and the stall is not. Add `gzip_compresslevel` (range 0-9). The default stays at 9, so nothing changes for existing installs; users who feel the stall on a large library can now lower it, and the docs explain when that is worthwhile. At 0 the middleware is left out entirely rather than installed at level 0, so responses skip the responder instead of being buffered and re-emitted as a stored-only gzip stream. Deployments behind a compressing reverse proxy want that. This does not help the media case — on incompressible input level 1 costs about the same as level 9 — which is why the content-type exclusion remains the fix for that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pfannkuchensack
force-pushed
the
perf/gzip-compresslevel
branch
from
August 2, 2026 03:49
5ac2346 to
c8bae5c
Compare
Collaborator
|
@Pfannkuchensack This is neat! I think it would be more descriptive for the config option to be something like |
The old name read like an image setting, easy to confuse with pil_compress_level. The new one says which layer it acts on. Renames the config field, its env var (INVOKEAI_HTTP_COMPRESSION_LEVEL), the docs, the tests and the generated schemas. No behaviour change: the default stays 9 and 0 still leaves the middleware uninstalled. The setting has never been in a release, so no deprecation alias is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JPPhoto
requested changes
Aug 8, 2026
JPPhoto
left a comment
Collaborator
There was a problem hiding this comment.
A few minor things:
-
PR QA text is stale: claims default
1and usesINVOKEAI_HTTP_COMPRESS_LEVEL; source defines default9andINVOKEAI_HTTP_COMPRESSION_LEVELininvokeai/app/services/config/config_default.py:168anddocs/src/generated/settings.json:119. Make sure the default is correct and documented properly. -
get_queue_item_summaries_by_ids()builds one SQLite placeholder per unbounded ID ininvokeai/app/services/session_queue/session_queue_sqlite.py:1365; requests above SQLite's variable limit return 500.Test:submit >32,766 IDs; expect chunking or 422, never 500.
Alternative implementation ideas:
-
Instead of one unbounded
IN (...), chunk IDs and merge by requested order; avoids SQLite-limit failures. -
Instead of mutating Starlette responder state after
super(), gate compression by media type before invoking gzip; reduces dependency on responder internals and upgrade regressions. -
Consider proxy/worker compression or default level 1; removes level-9 event-loop stalls.
…v var Addresses review feedback on invoke-ai#9441. - get_queue_item_summaries_by_ids built one bound parameter per requested id, so a request above SQLite's SQLITE_MAX_VARIABLE_NUMBER (32766 on modern builds, 999 on older ones) failed with an OperationalError and answered 500. Deduplicate the ids and query in chunks of 900, reassembling in the order the caller asked for. Follows the existing UserService.get_many pattern. - The config docs did not name the environment variable. `http_compression_level` is settable as INVOKEAI_HTTP_COMPRESSION_LEVEL; the default of 9 is unchanged and was already correct in the source and in the generated settings reference. - Pin the Starlette responder contract that _ContentTypeAwareGZipResponder depends on (the exclusion flag exists; http.response.start is buffered, not forwarded) so an upgrade that breaks it fails with a named cause instead of silently gzipping PNGs.
Member
Author
|
I have fixed the SQLite and renamed the config. I dont change the default here but fixed the docs for it. |
JPPhoto
self-requested a review
August 15, 2026 17:42
JPPhoto
approved these changes
Aug 15, 2026
JPPhoto
left a comment
Collaborator
There was a problem hiding this comment.
Approved for merging after 9436!
invoke-ai#9436 has landed in main, so most of what this branch carried is now upstream and the overlap had to be unpicked file by file. main wins wherever it evolved past what this branch forked from: - session_queue.py: the two sanitizers were consolidated into one generic sanitize_queue_item_for_user over a TypeVar, so the summary-specific name this branch called no longer exists. main also bounds item_ids with max_length, which is the 422 answer to the bind-limit review comment. - session_queue_sqlite.py: main implements the same chunked IN (...) lookup this branch added, and additionally selects parent_item_id. Its regression test reads the bind limit off the running SQLite build instead of hardcoding one, and interleaves real ids with padding, so it also covers chunks matching nothing. - model_manager.py: main serializes the HF token reset under _HF_TOKEN_LOCK. - blocking-work-in-api-routes.md: main documents the anyio thread limiter ceiling. - test_no_blocking_async_routes.py: main has two closure/await cases more. This branch wins for its own subject: configure_gzip, GZIP_MINIMUM_SIZE, the http_compression_level wiring in api_app.py and the twelve tests in test_gzip_content_types.py (a superset of main's five). openapi.json, schema.ts and uv.lock are taken from main after checking it is a superset - same 145 paths, one schema more, nothing only on this side - and 'uv lock --check' confirms the lockfile still matches the merged pyproject.
The main merge resolved openapi.json and schema.ts in favour of main, which drops this branch's own contribution: http_compression_level is a field on InvokeAIAppConfig, so it lives inside a schema's properties rather than adding a path or a schema of its own. Comparing the two files by path and schema *names* - which is how the merge was checked - cannot see a difference at that depth, so the loss went unnoticed until openapi-checks and typegen-checks failed. Restores the property and the InvokeAIAppConfig description that lists it. schema.ts is regenerated from the corrected openapi.json rather than hand-edited.
The previous regeneration ran the typegen script from a different checkout, which only maps FastAPI's pre-0.130 'format: binary' to Blob. Since the schema is now generated by FastAPI 0.141, upload fields arrive as 'contentMediaType: application/octet-stream' and came out as string, breaking StylePresetImportButton with TS2345 and leaving typegen-checks red. Regenerated with the script and the locked openapi-typescript from this branch. tsc --noEmit passes and a second run reproduces the file byte for byte.
lstein
enabled auto-merge (squash)
August 18, 2026 21:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
perf: GZip compression level for API responses is now configurable.
app.add_middleware(ContentTypeAwareGZipMiddleware, minimum_size=1000)never passed acompresslevel, so it inherited Starlette's default of 9 — the slowest setting. Compression runs entirely on the event loop, which means its cost is not paid by the requesting client alone: for its full duration no other request is served and no socket.io event is delivered.Measured on the flat image-name list of a 200,000-image library (8.48 MB of JSON, production-style UUID filenames):
Level 9 spends 5.5× the event-loop time to save 0.4 percentage points of bandwidth. For a locally-served app — the normal case — the saved bandwidth is worthless and the stall is directly felt. This matches the profile of the
item_namesendpoint, whose remaining ~102 ms p95 of loop blocking was almost entirely gzip.How:
http_compression_level: int = 9(ge=0,le=9) inInvokeAIAppConfig, settable asINVOKEAI_HTTP_COMPRESSION_LEVEL. The default is unchanged from the effective behavior before this PR, so upgrading does not silently alter anyone's install; the docs say when to lower it.configure_gzip(app, compresslevel)inapi_app.pydoes the wiring. At0the middleware is not installed at all rather than installed at level 0 — otherwise every response would still be buffered through the responder and re-emitted as a stored-only gzip stream, paying the overhead for no benefit.configuration/invokeai-yaml.mdx, including the env var and the reverse-proxy case.What this explicitly does not fix: lowering the level does nothing for the media path. On incompressible data (PNG/WebP/MP4 are already deflate-compressed) level 1 costs essentially the same as level 9, because deflate must scan the data either way — a 1024×1024 PNG takes 51.0 ms at level 1 vs 52.2 ms at level 9 and comes back larger than it went in. That case is fixed by the content-type exclusion, not by the level.
Related Issues / Discussions
Started as a follow-up to #9436 (
ContentTypeAwareGZipMiddleware— skip compression for already-compressed content types) and now carries that work as well, since this branch is targeted atmain.See also
docs/src/content/docs/contributing/blocking-work-in-api-routes.mdfor why work on the event loop stalls the whole process.QA Instructions
Automated:
159 tests pass. Compression coverage: no response carries
Content-Encoding: gzipwhenhttp_compression_level=0; the middleware is absent fromapp.user_middlewareat level 0; the configured level actually reaches the compressor (level 9 produces a smaller body than level 1 on a realistic name list); the real app uses the configured level; the default is 9; out-of-range levels are rejected at config validation; and the Starlette responder contract the content-type exclusion depends on is pinned so an upgrade fails with a named cause.Queue coverage (from review feedback):
get_queue_item_summaries_by_idschunks its id lookup, so a request of 40,000 ids returns normally instead of exceedingSQLITE_MAX_VARIABLE_NUMBERand answering 500; repeated ids still map one-for-one onto the request list.Manual:
curl -H 'Accept-Encoding: gzip' -sI http://127.0.0.1:9090/api/v1/images/namesreturnscontent-encoding: gzip.http_compression_level: 0ininvokeai.yaml(orINVOKEAI_HTTP_COMPRESSION_LEVEL=0) and restart. The same request must return nocontent-encodingheader, and the UI must still work.Accept-Encoding: gzipat the default level — it must come back withoutcontent-encoding.http_compression_level: 1vs the default9.INVOKEAI_HTTP_COMPRESSION_LEVEL=10must fail config validation.Merge Plan
Targeted at
mainand self-contained. No DB or redux-slice changes.Note that this branch also lifts the FastAPI pin from
==0.118.3to>=0.141.1,<0.142; the 0.119.0 OpenAPI crash that motivated the old pin was a FastAPI bug and is fixed as of 0.124.0.Checklist
What's Newcopy (if doing a release after this PR)