feat(architectures): serve model policy from the backend, and read it in webv2 - #146
Open
Pfannkuchensack wants to merge 40 commits into
Open
Conversation
- Added `scheduler_applies_to_graph` to ERNIE-Image architecture definition to ensure proper UI behavior. - Introduced a new fixture for architecture capabilities to maintain consistency between backend and frontend. - Enhanced generation policies to correctly prioritize guidance settings based on architecture labels. - Added tests to verify that the scheduler application flag aligns with the node's capabilities.
…ven-model-policies
… in webv2 `BASE_GENERATION` in webv2 held 14 architectures with grid, canvas, steps, CFG, scheduler, negative-prompt policy and six UI flags -- the same facts `architectures/defs/` declares. Adding a model type meant editing the frontend too, even when nothing about it was special. It is deleted; the policy now comes from `GET /api/v2/models/capabilities`. ## Reading the table `core/` cannot reach `data/` under `feature-core-purity`, so the table is pushed in through a registry, the shape `configureHttpAuth` already uses. `SupportedGenerateBase` is no longer `keyof typeof BASE_GENERATION` but a declared list beside `GRAPH_BUILDERS`: which architectures we can build a graph for is the one question only the frontend can answer, and it must not depend on load state -- `isSupportedGenerateModel` has a dozen callers outside the widget. Loading is gated on two levels. The widget shows a loading state, but the load-bearing gate is in `createGenerateWidgetSyncRuntime`: it reconciles at construction and *persists* the result into the project, so without the table it would write fallback defaults to disk -- an off-grid canvas for CogView 4 among them. `resolveGenerateWidgetValues` returns null until the table is there. Defaults change for six architectures, deliberately. FLUX moves to 28 steps at guidance 3.5 with real variant rows (Schnell 4, Fill 50 at guidance 30) -- the single row could not express that and carried Schnell's step count for every variant. sd-3 40/4.5, cogview4 50/3.5, anima 35/4.5, z-image 9 steps, SD 2.x 768px. Control kinds, reference-image limits and regional-guidance support are read from the table too; their data was verified identical to the hardcoded rules first, so those are behaviour-preserving. The limits in `controlValidation.ts` and the FLUX-Fill exception stay -- the endpoint has no column for them, and a comment says so. ## Which VAEs an architecture accepts The same fact was written down three times -- once per loader as `ui_model_base`, once in webv2's VAE picker, once more in its related-models linker -- and all three disagreed about Anima. It is not derivable: eight architectures share a 16-channel space at 8x compression, but SD 3.5 and CogView 4 belong to neither the Wan/Qwen family nor FLUX's. Nor does the VAE's own base settle it. The same physical file is registered as `anima` or `qwen-image` depending on which family it was installed for -- byte-identical, 194 tensors -- and a `wan` VAE is either the shared 16-channel Wan 2.1 file or TI2V-5B's 48-channel Wan2.2-VAE, which fits nothing else. So the new facet declares base *and*, where the base is ambiguous, the channel count `VAE_Checkpoint_Wan_Config` has been recording all along without anyone reading it. Two loader declarations were wrong and are corrected. `anima_model_loader` declared nothing, so its picker offered every VAE ever installed; it accepts the Wan-family file under any of its three bases plus a FLUX one, since `anima_l2i` branches on `isinstance(vae, (AutoencoderKLWan, FluxAutoEncoder))` with a separate decode path for each. `flux2_klein_model_loader` offered FLUX VAEs, claiming Klein "uses the same VAE as FLUX (16-channel)" -- it does not: generating with one raises `AutoEncoder.decode() got an unexpected keyword argument 'return_dict'`, the legacy 16-channel class against a 32-channel latent. ## Loading a Qwen-Image VAE Two layouts of that autoencoder are in circulation. Repo exports carry diffusers keys (`decoder.conv_in.weight`); community redistributions carry the original layout (`decoder.conv1.weight`) and need converting. Only the first was handled, and `AutoencoderKLQwenImage` registers no single-file conversion, so a redistributed file failed `load_state_dict(strict=True)` with all 194 keys missing. `strict=True` was right; the assumption in front of it was not. The byte-identical checkpoint installed under `anima` loaded fine all along through `AutoencoderKLWan.from_single_file`, which converts -- so whether a VAE worked came down to which base it happened to be probed as, and `VAE_Checkpoint_Wan_Config` deliberately hands 16-channel files to the Qwen-Image config unless the filename says "wan". The loader now looks at the state dict: diffusers keys take the direct path with `strict=True` unchanged, anything else is converted as the Wan-family file it is. No installation needs adjusting -- identification, model records and bases are untouched. ## Bugs that fall out of removing the duplicates - `bboxGrid.ts` kept its own copy of the grid column and had drifted: krea-2, wan and ideogram-4 got 8px steps while their denoise nodes enforce multiples of 16, so the canvas offered sizes that failed at enqueue. - `RegionalGuidanceSettings` asked "is this the FLUX family?" where the graph builder asks `regional_negative`. On krea-2 it rendered a negative prompt and an Auto-Negative switch that `addRegionalGuidance` then discarded. - ERNIE-Image's `FeaturesFacet` omitted `scheduler_applies_to_graph`, defaulting it to False, while `ernie_image_denoise` builds its sampler from the `scheduler` field. Thirteen of fourteen agreed; nothing compared them. - `getGenerationDefaults` folded `cfg_scale ?? guidance`, so a guidance-distilled model's "CFG off" marker won over the guidance it actually samples with. FLUX dev generated at guidance 1.0 instead of 3.5. ## Keeping it honest A committed fixture of the response body is pinned against `architecture_capabilities()` by a backend test, feeds the frontend's unit tests, and is what `scripts/mock-backend.mjs` serves -- so the hand-written wire type, the tests and the mock cannot drift from the real route. Invariant tests bind each facet to the node that enforces it: `dimension_grid` to `multiple_of`, `scheduler_applies_to_graph` to whether the denoise node takes a scheduler, and each `VaeFacet` to its loader's `ui_model_base`. Tests seed the registry through `architectureCapabilities.testing.ts` rather than a global setup file, so a test that wants to observe the unloaded behaviour still can. Verified end to end against real models: all seven installed architectures generate, previews included, and a `qwen-image`-registered VAE that failed before now works.
…erver-driven-model-policies
Four CI jobs failed, from two causes. `oxfmt --check` rejected 28 files. They were linted and typechecked but never formatted: on a Windows checkout `core.autocrlf` gives every file CRLF while oxfmt writes LF, so `format:check` fails for all 1898 files locally and is useless as a signal. Formatted the ones this branch touches. `openapi.json` and `schema.ts` were stale. Correcting `ui_model_base` on the Anima, Qwen-Image and FLUX.2 Klein loaders changes the invocation schema, and the artifacts were last regenerated before that. Two more surfaced once formatting stopped short-circuiting the lint job: - The test helper was named `useArchitectureCapabilitiesFixture`, so `react-hooks/rules-of-hooks` read it as a hook called at module top level. It is not one; `seedArchitectureCapabilities` says what it does anyway. - Dead imports left behind by deleting `BASE_GENERATION` and moving the ordered-bases pin out.
Four CI jobs failed, from two causes. `oxfmt --check` rejected 28 files. They were linted and typechecked but never formatted: on a Windows checkout `core.autocrlf` gives every file CRLF while oxfmt writes LF, so `format:check` fails for all 1898 files locally and is useless as a signal. Formatted the ones this branch touches. `openapi.json` and `schema.ts` were stale. Correcting `ui_model_base` on the Anima, Qwen-Image and FLUX.2 Klein loaders changes the invocation schema, and the artifacts were last regenerated before that. Two more surfaced once formatting stopped short-circuiting the lint job: - The test helper was named `useArchitectureCapabilitiesFixture`, so `react-hooks/rules-of-hooks` read it as a hook called at module top level. It is not one; `seedArchitectureCapabilities` says what it does anyway. - Dead imports left behind by deleting `BASE_GENERATION` and moving the ordered-bases pin out.
…hub.com/invoke-ai/InvokeAI-7 into feat/webv2-server-driven-model-policies
`compileUpscaleGraph` passes Upscale's own `kdpm_2` through `coerceSchedulerForGraph`, which reads architecture policy. Without the table the fallback config applies, and its `schedulerAppliesToGraph: false` makes the function return the fallback scheduler rather than pass the requested one through -- so the compiled graph carried `euler_a`. Missed because the webv2 suite was run in subsets while iterating. It completes in under two minutes; there was never a reason not to run all of it.
…test `deriveImageRecallCapabilities` reports whether an image's metadata can be recalled, and clip skip depends on the architecture's `clipSkipMax`. That comes from the capability table now, so without it the fallback applies and the capability reads false. Missed because `vitest.config.mts` excludes `*.browser.test.*` -- they run under `vitest.browser.config.mts`, which had not been run at all. Both suites pass now: 807 browser tests and 6435 node tests.
Reading architecture policy from the backend adds four modules to the editor and launchpad bundles: +52 bytes owned JavaScript on launchpad, +281 on editor, and one additional script request per editor route. Only the structural numbers are updated. `--update-baseline` also re-records timings, and those come from whichever machine ran it -- the fresh numbers were ~10% above the committed ones, which would have loosened a gated tolerance and let real regressions of that size through. The 56 timing values are kept as committed.
`--update-baseline` writes `architecture-baseline.json` itself and not in oxfmt's style, so `format:check` rejected it. Values are unchanged -- the only difference is how one array is wrapped.
`oxfmt` and `prettier` read the file on disk, so on a Windows checkout `core.autocrlf=true` makes `format:check` fail for all 1898 files -- including every file the contributor never touched. The check is therefore unusable locally, and formatting errors are only found by CI. That happened four times in a row on one PR. `* text=auto` already normalizes these files to LF in the repository, so this changes the working copy only: `git add --renormalize invokeai/frontend` reports nothing, and no file content differs after re-checkout. `docker/**` has carried the same rule for the same reason. Verified after re-checkout: `pnpm run format:check` reports "All matched files use the correct format", and `prettier --check` passes for the legacy frontend.
…erver-driven-model-policies The base grew a regional-guidance support matrix covering eight bases while the FeaturesFacet still declared five; that only becomes visible once the frontend reads the policy from the backend, so sd-2, z-image and anima now declare supports_regional_guidance (regional_negative for sd-2 alone), and the capabilities fixture and the two declaration guards follow. The backend answers whether a base supports regional guidance; the matrix supplies the graph knowledge that answer selects. Guards pin that the two agree on regional negatives and that every base declared supported has a matrix row, since the type narrowing now rests on the backend's answer. Re-measured architecture-baseline.json for the merged bundle: this branch's four capability modules enter the editor's initial graph, +281 B owned JS (+0.21%) and one request. browser-baseline.json still needs a capture on the reference machine; its timings and browser path are machine-specific.
…erver-driven-model-policies
The generator emits raw json.dumps output; both openapi-checks and the frontend prettier check compare against a prettier-formatted file.
…erver-driven-model-policies Four architecture defs gained a facet on each side -- VaeFacet here, VariantFacet on the base -- and both are kept. Regenerated openapi.json and schema.ts.
7 tasks
…erver-driven-model-policies Both branches had found and fixed the same two things independently; the merge keeps this branch's versions, which are the better-structured ones: - the guidance/cfg_scale choice, as `getRecordGuidanceValue` keyed on `guidanceLabel` rather than an inline ternary - ernie-image's `scheduler_applies_to_graph` Removed as obsolete here: #133's `capabilityContract.test.ts`, its snapshot and `test_frontend_capability_parity.py`. They pinned webv2's `BASE_GENERATION` to the registry, and this branch deletes that table in favour of reading the capabilities endpoint — `test_capabilities_fixture.py` is the contract now, in the other direction. Their oxfmt exclusion went with them. Kept from #133 because this branch does not have it: the guidance slider could not hold a value above its 10-wide track, so FLUX Fill's 30 clamped to 10 on first blur. Its regression tests were folded into this branch's own guidance describe block, dropping the three cases already covered there and re-pointing the per-base sweep at the policy instead of the deleted table.
Serving generation policy from the backend puts four modules in the app shell's static graph, and this branch updated architecture-baseline.json for it but not browser-baseline.json. Re-measured: - otherRawBytes 152115 -> 153662 on every route. That bucket is exactly dist/locales/en.json plus dist/index.html; the growth is the three strings the capabilities fetch needs for its loading, error and retry states. - requestCount and scriptRequestCount +1 on the four editor routes, plus ~15 KB of scriptRawBytes, for the store and its two helpers. launchpad is unchanged -- there the modules fold into chunks it already requests. - activatedResourceLimits.largestAssetRawBytes tightened 322887 -> 310251. editor-gallery now loads a table it never reads. That is real waste, but it is the debt those routes already carry as `deepen-widget-registry-loading`, not a new category, so it is left to that ticket rather than reshaped here. Captured on Windows: the enforced resource numbers reproduce the CI runner's byte for byte, but browserExecutable and the timing medians are this machine's. Timings are not gated (timingPolicy.enforce is false, runner unconfigured); re-capture on the reference machine before turning them on.
7 tasks
…erver-driven-model-policies
The capability gate covered the Generate widget's resolver, so the canvas and topbar Invoke paths still compiled against FALLBACK_GENERATION_CONFIG — a grid of 8 for bases that reject anything but 16, which is the enqueue failure this feature exists to prevent. Moved to `getGenerationValidationReasons`, the one function graph.ts, compileCanvasGraph.ts and previewGraph.ts all call, and placed before the unsupported-model check so a loaded-late table no longer reports itself as "needs a supported model". Three workbench suites seed the table now, as app boot does. `VaeFacet` had no production reader — `capabilities.py` never rendered it, so the fact never reached a client and webv2's own copies could not be bound to it. It is served now, and a test drives every declared pairing through the picker's filter. That immediately surfaced two desyncs this PR introduced by widening the backend lists alone: qwen-image accepts an anima-registered VAE, and anima accepts the 16-channel wan file — the same 194-tensor checkpoint under whichever base it was installed. Both are the picker refusing a VAE the backend would load. `isRegionalGuidanceSupportedForBase` narrowed to the key set of a hardcoded matrix while deciding from the backend's answer, so a base declared supported with no row returned `undefined` typed as a support object. The guard written to catch that used `not.toBeNull()`, which `undefined` passes. Both fixed; the drift now fails six tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…licy - Every capability reader subscribes to the table instead of latching the fallback: the canvas grid, the dims sync, the topbar, regional guidance and gallery import. Reads happen inside the store selector -- React Compiler memoises a bare policy call in a component body for the component's life. - The Generate widget resolves again after a successful retry, and its loading and error states are announced and keep focus on the retry button. - Anima no longer offers FLUX VAEs: same channel count, different basis, and the decode is measured at 6.10 dB PSNR against 46.18 for the correct one. - The Qwen VAE branch dispatches on positive layout markers, loads in bfloat16 without a network round trip, and raises on a checkpoint that is neither. - webv2 reads VAE compatibility from the served table rather than a fourth hand-written copy, and joins the dimension grid on (base, variant).
…erver-driven-model-policies Both sides had rewritten the same declarations, so three conflicts needed merging rather than picking: capabilities.py renders the VAE policy and the per-variant grid together, and the two VaeAcceptance models were opened up like the four the base branch already had -- a response model gains nothing from extra="forbid" but stamps the schema shut. The performance baselines were re-recorded from a build of the merged tree. Neither side's numbers were right: the merge carries both sets of changes and measures above both, so taking either would have under-budgeted the gate. test_frontend_capability_parity.py stays deleted. It compared the backend against a frontend table this branch removes; test_capabilities_fixture.py checks the served payload by equality instead, and validate() already refuses to boot an architecture that is missing.
…erver-driven-model-policies
Recalled dimensions are snapped to the architecture's grid and then written into the project. Without the served table every base reads as grid 8, so a krea-2 or CogView 4 project stored a size its own denoise node rejects, and nothing re-derived it. Both size helpers now fail closed, the way the resolver and the canvas gate already do, so the recall reports no size instead. The blocking reason no longer says capabilities have "not loaded yet": after a failed fetch nothing re-kicks it except the Generate panel's retry, and `core` cannot see the store that tells loading from failed.
Pfannkuchensack
marked this pull request as ready for review
September 12, 2026 01:09
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant and
lstein
as code owners
September 12, 2026 01:09
…erver-driven-model-policies
…erver-driven-model-policies
The picker stopped offering FLUX VAEs, but the nodes still accepted one: a hand-built graph, a saved workflow or a metadata recall reached the same path. Measured on a real denoise -- one latent, two decodes in one graph -- the FLUX decode scores 8.67 dB PSNR against the correct one, mean absolute error 84 of 255, and the run still reports success. Both nodes now refuse before the model reaches a device, and the message names the VAEs that do work. The encode side had the same gap in reverse: it would have handed the denoiser a latent in the wrong basis.
…erver-driven-model-policies
The guidance input allowed 0 to 100 for every model, so a value the UI offered was rejected at enqueue: FLUX.2 accepts at most 20, and four architectures reject below 1. The bounds now come from the served capability row, joined on (base, variant) like the dimension grid. FLUX.1 Fill still types and keeps 30 -- its node is genuinely unbounded, and the local cap stays where the architecture declares no ceiling. A stored value is repaired on model selection, beside the existing CLIP-skip clamp, and validated at enqueue for everything that does not pass through selection -- a reopened project, a metadata recall, settings written by API. Ideogram 4's own controls were bounded by hand and all three were wrong: mu accepts -4..4 where the UI offered 0..10, so the whole negative half was unreachable and everything above 4 was rejected. Out-of-range slider marks are dropped rather than clamped -- a marker pinned to the end of the track claims a default that is not there. The model-manager form capped guidance at 20, which this branch made unreachable: FLUX.1 Fill now declares 30, so opening such a model and saving any field was refused over a value the user never typed. The form mirrors the record's own bound instead.
The previous commit staged the file while it was being regenerated, so it landed empty. The content is unchanged from the merge commit before it: the guidance bounds reached the schema with the backend half, and nothing in that commit touches the API surface.
…erver-driven-model-policies - Re-record webv2 performance baselines in the delta-gated schema from main - Seed the capability table in the recall tests added on main
- Resolve the invocation route inside the capabilities selector so React Compiler cannot serve the stale result - Keep the models mock snapshot stable so the regression test fails without the fix
…e-aware VAE loading - Picker, validation and every graph builder (incl. SD and SD3) read VAE compatibility from the served row - VAE loader reads pickled checkpoints and keeps float32; only float16 is raised to bfloat16 - External generators keep the generic dimension grid for recall and canvas sync
…VAEs - Drop modality and spatial_compression from the capabilities table; read regional_negative from it - VaeFacet.by_variant: Wan A14B takes the 16-channel VAE, TI2V-5B the 48-channel one - Canvas scaling, control layers and Invoke re-read policy when capabilities or node templates load
…bility outage states - Anima VAE nodes accept both Wan 2.1 VAE layouts; loaders derive VAE bases and validation from VaeFacet - Z-Image Base scheduler set declared per variant; Ideogram 4 bounds pinned to the node - Control layers and the Generate retry handle a missing capability table without false warnings or lost focus
- as_qwen_image_vae and the tiling scope live in backend/util: Qwen-Image, Krea-2 and Anima all use them - Anima tiling test builds its tiny VAE with the geometry the node accepts - focusIfUnclaimed joins the widget-hosts chunk and is pinned in the editor source-owner sets
- Recall All, Remix and CLIP skip wait for the capability table; prompts and seed stay recallable - capability retries settle through the load promise instead of useEffect subscriptions - the Wan VAE loader test loads a real checkpoint through strict state-dict validation
…ture' into feat/webv2-server-driven-model-policies
…bsent - API recall, image recall onto a fresh project and queue recall no longer compute or persist fallback policy; prompts and seed still apply - Reference-image availability re-reads when the table lands instead of latching its first answer - Outage notices come from the locale catalog
Place Workflow before routing checks and select an overflowing filmstrip list. Match the Settings menu label with its shortcut suffix.
Collaborator
|
🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adding a
BaseModelTypeshould not mean editing the frontend. It still did:BASE_GENERATIONin webv2 held 14 architectures with grid, canvas, steps, CFG, scheduler, negative-prompt policy and six UI flags — the same factsinvokeai/backend/architectures/defs/already declares. This deletes that table and reads the policy fromGET /api/v2/models/capabilitiesinstead.Three things had to happen for that to be safe, and each turned up a bug of its own.
Reading the table.
core/cannot reachdata/underfeature-core-purity, so the rows are pushed into a registry — the shapeconfigureHttpAuthalready uses.SupportedGenerateBasebecomes a declared list besideGRAPH_BUILDERSrather thankeyof typeof BASE_GENERATION: which architectures we can build a graph for is the one question only the frontend can answer, and it must not depend on load state, sinceisSupportedGenerateModelhas a dozen callers outside the widget.Loading is gated twice. The widget shows a loading state, but the load-bearing gate is in
createGenerateWidgetSyncRuntime, which reconciles at construction and persists its result into the project — without the table it would have written fallback defaults to disk, including an off-grid canvas for CogView 4.Which VAEs an architecture accepts. The same fact was written down three times — once per loader as
ui_model_base, once in webv2's VAE picker, once more in its related-models linker — and all three disagreed about Anima. It is not derivable: eight architectures share a 16-channel space at 8× compression, but SD 3.5 and CogView 4 belong to neither the Wan/Qwen family nor FLUX's. Nor does the VAE's own base settle it — the same physical file is registered asanimaorqwen-imagedepending on which family it was installed for (byte-identical, 194 tensors), and awanVAE is either the shared 16-channel Wan 2.1 file or TI2V-5B's 48-channel one. So the new facet declares base and, where the base is ambiguous, the channel countVAE_Checkpoint_Wan_Confighas been recording all along without anyone reading it.Loading a Qwen-Image VAE. Two layouts of that autoencoder circulate; only the diffusers one was handled, with
strict=True, so a community redistribution failed with all 194 keys missing — while the byte-identical file installed underanimaworked, because that path converts. Whether a VAE worked came down to which base it happened to be probed as.Defaults change for six architectures
Deliberate, and the reason this is a
featand not arefactor. webv2 is not shipped yet, so no existing user is affected.The variant rows are the point: one row per architecture could not express that FLUX Schnell wants 4 steps where dev wants 28, so it carried Schnell's number for every variant.
Bugs fixed on the way
bboxGrid.tskept a second copy of the grid column and had drifted — krea-2, wan and ideogram-4 got 8px steps while their denoise nodes enforce multiples of 16, so the canvas offered sizes that failed at enqueue.mainbefore this branch, and the only change toRegionalGuidanceSettingsbelow is a comment. What this branch does change there is the unloaded state: it no longer claims a supported model is unsupported while the table is in flight.flux2_klein_model_loaderoffered FLUX VAEs, claiming Klein "uses the same VAE as FLUX (16-channel)". It does not — generating with one raisesAutoEncoder.decode() got an unexpected keyword argument 'return_dict'.anima_model_loaderdeclared noui_model_baseat all, so its picker offered every VAE ever installed.scheduler_applies_to_graph, defaulting it toFalse, whileernie_image_denoisebuilds its sampler from theschedulerfield. Thirteen of fourteen agreed; nothing compared them.getGenerationDefaultsfoldedcfg_scale ?? guidance, so a guidance-distilled model's "CFG off" marker won over the guidance it actually samples with — FLUX dev generated at 1.0, not 3.5.How it stays honest
A committed fixture of the response body is pinned against
architecture_capabilities()by a backend test, feeds the frontend's unit tests, and is whatscripts/mock-backend.mjsserves — so the hand-written wire type, the tests and the mock cannot drift from the real route. webv2 has no generated OpenAPI types, so without this the TS type would be an unguarded assertion.Invariant tests bind each facet to the thing that enforces it:
dimension_gridto the node'smultiple_of,scheduler_applies_to_graphto whether the denoise node takes a scheduler, eachVaeFacetto its loader'sui_model_base, andnegative_prompt: nevertocfg_scale == 1.0. Four of the six bugs above were found by writing those, not by review.Related Issues / Discussions
Stacked on #133 (the architecture facet registry), which introduces the endpoint this consumes.
QA Instructions
Nothing about an existing install needs adjusting — no migration, no re-identification, no model records touched.
Generation. Open Generate and confirm the panel renders after a brief load. Pick FLUX Schnell and FLUX dev in turn: they should prefill 4 and 28 steps respectively, which the old single-row table could not do. Check SD 2.x prefills 768×768.
The grid fix. On the canvas with a Krea-2, Wan or Ideogram-4 model, the bbox should snap to 16px. Before this, 8px was offered and an off-grid size failed at enqueue — try 1032px wide on
mainto see the old behaviour.Regional guidance. Add a regional guidance layer with a Krea-2 model. The negative prompt field and Auto-Negative switch should be gone; they were rendered and then discarded by the graph builder.
VAE pickers. With an Anima model, the VAE list should offer Anima, Qwen-Image and 16-channel Wan VAEs and nothing else — no FLUX VAE, which decodes an Anima latent into noise; it previously offered every VAE installed. A Wan TI2V-5B model should offer only the 48-channel Wan VAE, an A14B model only the 16-channel one. With FLUX.2 Klein, FLUX VAEs should no longer appear; selecting one on
maincrashes during decode.The Qwen VAE fix. If you have a community Qwen-Image VAE registered under base
qwen-image(thedecoder.conv1.*layout rather thandecoder.conv_in.*), generating with it failed before with "194 missing keys" and should now work.The gate. Point the frontend at a backend without the
/capabilitiesroute, or make it 500. The Generate panel must show an error with a retry, not render fields prefilled with generic defaults.Verified end to end on this branch: all seven installed architectures generate with previews;
qwen-imagewith both VAE registrations; FLUX.2 Klein with the FLUX.2 VAE (and confirmed failing with the FLUX one).Merge Plan
Merge #133 first. This branch is stacked on it and consumes the endpoint it adds.
The diff currently shows ~96 files, but only 60 are this branch's — the rest is the generate-widget redesign merged in from
mainand will disappear from the diff once the base branch catches up.Nothing touches a DB schema, a redux slice or a migration.
openapi.jsonandschema.tsare regenerated, and the delta is larger than additive:ArchitectureVae,VaeAcceptance), one optional field on the capability row, andflow-no-lcminSchedulerSet.modalityorfeatures.spatial_compression; webv2, the only client, never read them.flux2_klein_model_loader'sui_model_baselosesflux, andanima_model_loader's loses it too, whileqwen_image_model_loadergainsanima. Legacyinvokeai/frontend/webfilters its VAE pickers on that field, so those pickers change for the shipping client. Both are the intended correction — Klein cannot decode with a FLUX VAE, and Anima decodes one into noise — but they are a behaviour change, not a no-op.Review follow-up
An adversarial review of this branch, plus two independent passes, produced findings that are fixed
here. The theme: moving policy behind an endpoint means every consumer needs a real subscription,
and only two modules had one.
Invoke gate, regional guidance, gallery import and image recall. Each previously read the registry
imperatively and latched the fallback for the rest of the session. Reopening a saved krea-2
project before the table arrived left the bbox snapping at 8px all session, on an architecture
whose denoise node enforces 16.
gridSizeForModelBase(base)in a component body is a pure function ofbaseto the compiler andis memoised for the component's lifetime. Policy reads happen inside the store selector now.
useMemodid not depend oncapability readiness, so the fail-closed
nullcached during the error render survived the retryand the form came back with an empty model picker for a project that had one saved.
architecture's grid and written into the project; without the table every base read as grid 8.
anima_latents_to_imagetakes a
FluxAutoEncoderwithout raising and skips the Wan denormalisation. Measured on realweights — one Anima denoise decoded by both VAEs, 8.67 dB PSNR between the two images, a magenta
moire in place of the subject, and no error or log line. The number is recorded in the
declaration, the loader field and the invariant test so it does not come back "as a fallback".
directly (in the configured precision, with float16 raised to bfloat16) rather than reaching for a hub config over HTTP, and raises on a checkpoint
that is neither layout instead of returning a partially-initialised model.
switchoverliteral base lists, which is what the new facet was for.
role="status"/role="alert"), the retry button keepskeyboard focus, and eleven test files now observe the unseeded state — previously every test
seeded the table before render, which is why none of the above was caught.
A second review round found three more, all the same shape as the VAE one — a control offering
values the graph rejects:
architectures reject below 1, so the UI offered values that failed at enqueue. The bounds are
declared per architecture, pinned against the
ge/leof the field each denoise node actuallyvalidates, and served on base and variant rows. FLUX.1 Fill keeps 30 — its node is unbounded.
muaccepts -4..4where the control offered 0..10: the entire negative half was unreachable and everything above 4
was rejected.
stepsstarts at 2, not 1.now declares 30, so opening such a model and saving any field was refused over a value the user
never typed. That one is a regression this branch introduces: before it, FLUX models carried no
stored defaults at all.
Out-of-range slider marks are dropped rather than clamped; a marker pinned to the end of the track
claims a default that is not there.
Still open, deliberately: a failed capability fetch is retried from the Generate panel or a control
layer's settings, not automatically.
Third review round
Findings from two further review passes, fixed here. The theme is the same as before, one level further down: a fact the backend declares should have one owner, and every place that repeated it -- in webv2, in loader field declarations, in node guards -- now reads it.
The capability table serves only what a client reads.
modality(modes,metadata_slug) andfeatures.spatial_compressionhad no reader in webv2 and are gone from the response.features.regional_negativehad a hand-kept copy inREGIONAL_GUIDANCE_SUPPORT; the regional graph now reads the served value.VAE compatibility is per variant, and the backend is its only owner.
VaeFacet.by_variantdeclares the split; theti2v_5brow carries its own list, and a test checks every variant's accepted VAEs against the latent space that variant denoises in.anima-registered VAE the picker offered; SD3 used to send a stale VAE the picker hid.ui_model_basefrom the facet (accepted_vae_bases), and the Wan and Krea-2 loaders validate a standalone VAE throughaccepts_vae, with the same actionable messages.Anima VAEs. Anima's encode and decode now accept the Wan 2.1 VAE in both layouts -- the original layout loads as
AutoencoderKLWan, the diffusers export of the Qwen-Image VAE asAutoencoderKLQwenImage-- where the table said yes and the nodes raised. A FLUX VAE is still refused with a message naming what to choose instead, and Wan 2.2's 48-channel VAE is now refused before it is moved to the device.anima_i2l1.0.2,anima_l2i1.0.4. The helpers that decide which VAE fits (as_qwen_image_vae, the tiling scope) moved frombackend/krea2/vae_compat.pytobackend/util/qwen_image_vae.py: Qwen-Image, Krea-2 and Anima all depend on them, and a file under one architecture hid that a change there changes all three. The Anima tiling test from #241 now builds its tiny VAE with the geometry the node accepts (16 channels, 8x) instead of one the guard rejects. One measurement is quoted everywhere: an Anima denoise decoded by both VAEs, 8.67 dB PSNR between the two images.VAE loading. Pickled checkpoints (
.pt,.pth,.ckpt,.bin) load again for the anima, qwen-image and wan registrations, and a float32 precision request is kept; only float16, which is unstable on this autoencoder, is raised to bfloat16. Thewanregistration's test loads a real (narrow)AutoencoderKLWancheckpoint through the loader's strictload_state_dict; its mock used to accept a key layout the loader would reject.Schedulers. Z-Image Base's "no LCM" rule was special-cased twice in webv2 against a
flow-no-lcmset the backend never served. It is declared per variant (scheduler_set_by_variant) and pinned toz_image_denoise's own description. Ideogram 4's steps / guidance / mu bounds stay frontend constants -- they belong to controls only that model has -- and a test reads them against the node'sge/le.States while the table is missing.
<body>.ensureArchitectureCapabilitiesLoadednow returns instead ofuseEffectsubscriptions; the control-layer panel hands focus over from a ref cleanup on its failure surface.Also in this branch
.gitattributes(commit7df4aac27f) checks frontend sources out with LF regardless ofcore.autocrlf. Without itformat:checkfails for every file on a Windows checkout, so formatting errors only surfaced in CI. It is a checkout setting only; the repository content is unchanged.architectureCapabilities.ts,supportedBases.ts, the capabilities API client and store) now load with the editor. The launchpad grows by 131 brotli bytes. The focus helper shared by the Generate panel and control-layer settings (focusIfUnclaimed.ts) is grouped into thewidget-hostschunk, so it adds no request; it is pinned in the three editor source-owner sets and stays out of the launchpad.Contract changes for the generated schema
openapi.jsonandschema.tsare regenerated. Not additive: the capability row losesmodalityandfeatures.spatial_compression, andSchedulerSetgainsflow-no-lcm.ui_model_baseon the krea-2 and qwen-image loaders changes order only (anima,qwen-image).Checklist
What's Newcopy (if doing a release after this PR)🤖 Generated with Claude Code