fix: close 34 findings from the PR 171-194 review - #241
Merged
Conversation
Tiling a diffusers VAE assembled a wrong-sized image for most legal tile sizes — in Z-Image, and pre-existing in SD1/SDXL decode and encode. Overlapping SDPA windows permanently corrupted torch's process-global backend selection. The int8 loaders cast integer payloads, left precision-sensitive layers quantized, and accepted mixed-format checkpoints that then loaded unscaled. `db_synchronous: normal` applied even without WAL, the one condition that makes it safe. Behaviour changes: - `force_tiled_decode` now applies to FLUX.1, which ignored it; a tiled decode is not pixel-identical to a single-pass one. - `INVOKE_KREA2_SDPA_BACKEND=priority-cudnn` is now `default` — it never ranked cuDNN first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant and
lstein
as code owners
September 11, 2026 15:09
`force_tiled_decode`'s description changed; `settings.json` is generated from it by a different script than the OpenAPI artefacts and was missed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to PRs #171, #172, #173, #174 and #194. An adversarial review of that merge window turned up 33 findings; this closes all of them, plus one pre-existing bug of the same class that the review exposed in SD1/SDXL.
The five that ship a wrong result rather than a crash
Tiling a diffusers VAE returned a wrong-sized image.
scoped_vae_tilingsettile_sample_min_sizeandtile_latent_min_sizeindependently, but diffusers derives its latent loop step from one and its output crop from the other — they only agree when the latent tile is a multiple of 4. The node field allows any multiple of 8. Measured on a 4-blockAutoencoderKLat a 768px target, sweeping all 84 legal values: 24 correct, 59 wrong-shaped, 1 crash. The same defect is pre-existing inlatents_to_image.py/image_to_latents.py: at a 512px target, 12 correct, 35 wrong-shaped, 2 crashes, and it affects encode as well as decode. Both now derive the latent tile first and the sample size from it, so the pair cannot disagree.Overlapping SDPA windows permanently corrupted process-global state.
torch.nn.attention.sdpa_kernelsaves and restoresat::globalContext()non-atomically, so two threads whose windows overlap leave it changed for the process lifetime — reproduced: priority order[FLASH, EFFICIENT, MATH, CUDNN]→[FLASH, CUDNN, EFFICIENT, MATH], and with an exclusive override, three backends left disabled. Both the Krea-2 attention processor and the Anima denoise loop open such windows, and InvokeAI runs one session per GPU concurrently, so it leaks into every other model in the process. Newinvokeai/backend/util/sdpa_scope.pygives the globals one owner: the outermost window is the only one that writes them.cast_unquantizedwidened integer payloads. The function the int8 loaders switched to lacked the float guard whose own comment warns that "a loader that switches tocast_state_dictdoes not silently lose the guard". An index buffer installed as a float raises inside the forward; a bool mask changes meaning silently.The int8 branch ignored each model's declared precision-sensitive modules.
ZImageTimestepEmbedderreadsself.mlp[0].weight.dtypeto pick its activation dtype; on anInt8ConvrotLinearthat istorch.int8, and the fallback looks for acompute_dtypeattribute neither class defines — so the timestep branch would run in float32 and hand a float32 embedding to a bf16 model.db_synchronous: normalwas applied without WAL.PRAGMA journal_mode = WALreports the mode it actually established and the return value was discarded. On a rollback journal,normalcan corrupt the database — the opposite of the documented guarantee. It is now read back, andnormalrefused for a database file that did not get WAL.Verified against real checkpoints
Every int8 fixture in the original PR was synthetic. Both published Comfy-Org int8 builds were loaded through the production loaders:
make_roomreservedInt8ConvrotLinear/ denseA real 1024×1024 forward through all 270 swapped Z-Image layers: 580 ms, 494 MiB activation peak, output finite, std 1.27 — the RAM prediction is exact on real files, and the layers compute.
Behaviour changes
force_tiled_decodenow applies to FLUX.1, which ignored it. A tiled decode is not pixel-identical to a single-pass one — the decoder's GroupNorms and mid-block attention are global, so the difference is spread across the image (measured 0.11 at 1024px on a ±2.7 range), not confined to the seams. The setting's description now says so.INVOKE_KREA2_SDPA_BACKEND=priority-cudnnis now spelleddefault. It never ranked cuDNN first — the shipped list is flash-first. An old exported value now fails at startup rather than mid-generation.int8_tensorwisewith scaled fp8 is refused at load instead of loading the fp8 weights unscaled. Scoped to modules the built model actually consumes, so an all-in-one export bundling a foreign submodel still loads.Verification
pytest -n logical: 7556 passed, 169 skipped, 9 xfailed. 9 failures are environmental on the dev box (HF_ENDPOINTpoints at a private-network mirror the SSRF guard refuses); none of the failing paths are touched by this diff.ruff checkandformat --checkclean;openapi.jsonandschema.tsregenerated.Known limitations
peak_int8_dequant_transient_bytesis exact for most shapes and ~2 MiB short on the two where cuBLAS takes a workspace, against a 3 GiB reservation floor. Documented at the function.