Update to transformers 5.1.0 by 4pointoh · Pull Request #8859 · invoke-ai/InvokeAI

4pointoh · 2026-02-07T01:14:22Z

Summary

This updates to Transformers 5.1.0. Transformers 5.0 adds support for the SAM3 model. My end goal is to add support for SAM3 in Invoke.

I have performed this update with AI assistance, I am certainly out of my depth with evaluating the true impact of these changes. However, I have produced a test plan with the help of AI and validated every scenario in the test plan. See the test plan at the bottom of this PR, which details every change and the reasoning for it.

NOTE: There is one dependency of InvokeAI which does NOT support Transformers 5.0+ -> The Compel project, which is used for prompt weighting: https://github.com/damian0815/compel

I have opened an issue on their repo here to bump their version: damian0815/compel#128

For now, I have added Transformers 5.1.0 to override-dependencies - And faced no issues after doing so. I have evaluated prompt weighting (which can be hard to validate by eye) and it appears to still work properly within my tests.

The most substantial code in this PR is the update to how the huggingface session is handled. My understanding, is that this new code only applies to test cases due to deprecation of a module which Invoke was using to produce mock callouts in the test environment, and that it has no effect on production functionality.

Related Issues / Discussions

QA Instructions

I have followed the testing plan at the bottom of this document. The testing plan was produced with AI assistance.

Merge Plan

Checklist

The PR has a short but descriptive title, suitable for a changelog
Tests added / updated (if applicable)
❗Changes to a redux slice have a corresponding migration
Documentation added / updated (if applicable)
Updated What's New copy (if doing a release after this PR)

Change Description & Test Plan (Generated with AI Assistance)

Regression Testing Plan: Transformers 5.1.0 + HuggingFace Hub Migration

Below is a change-by-change plan. Each section explains what changed, why, and how to test it. Tests are ordered from quickest smoke tests to longer end-to-end runs.

Change 1: `CLIPFeatureExtractor` → `CLIPImageProcessor`

File: invokeai/backend/stable_diffusion/diffusers_pipeline.py
Why: CLIPFeatureExtractor was removed in transformers 5.x in favour of CLIPImageProcessor.

#	Test	How
1a	Import smoke test	`python -c "from invokeai.backend.stable_diffusion.diffusers_pipeline import StableDiffusionGeneratorPipeline"` — should not raise `ImportError`
1b	SD 1.5 text-to-image	In the UI, load any SD 1.5 model and generate an image with a simple prompt (e.g. `"a cat on a couch"`). Confirm the image generates without errors. This exercises the full `StableDiffusionGeneratorPipeline` including the feature extractor type.

Change 2: `AutoFeatureExtractor` → `AutoImageProcessor` + removed `safe_serialization`

File: invokeai/backend/image_util/safety_checker.py
Why: All vision FeatureExtractor classes were removed in transformers 5.x. The safe_serialization parameter was also removed from save_pretrained (safetensors is now the only format).

#	Test	How
2a	Import smoke test	`python -c "from invokeai.backend.image_util.safety_checker import SafetyChecker"`
2b	NSFW checker first-time download	Delete the local cache at `<root>/models/core/convert/stable-diffusion-safety-checker/` if it exists. Enable the NSFW checker in config (`nsfw_checker: true`). Generate any SD 1.5 image. Confirm the safety checker downloads, saves to disk (no `safe_serialization` error), and the image either passes or is correctly blurred.
2c	NSFW checker cached load	With the cache from 2b still present, generate another image. Confirm it loads from local path via `AutoImageProcessor.from_pretrained()` without re-downloading.

Change 3: `T5TokenizerFast` → `T5Tokenizer` (4 files)

Files:

invokeai/app/invocations/flux_text_encoder.py
invokeai/app/invocations/sd3_text_encoder.py
invokeai/backend/model_manager/load/model_util.py
invokeai/backend/model_manager/load/model_loaders/flux.py

Why: Transformers 5.x unified slow/fast tokenizers — T5TokenizerFast no longer exists as a separate class. T5Tokenizer now uses the Rust backend by default.

#	Test	How
3a	Import smoke tests	Run each: `python -c "from invokeai.app.invocations.flux_text_encoder import FluxTextEncoderInvocation"`, same for `sd3_text_encoder`, `model_util`, and the flux loader module.
3b	FLUX text-to-image	Load a FLUX model (e.g. FLUX.1-dev or schnell). Generate an image with prompt `"a lighthouse on a cliff at sunset"`. This exercises `T5Tokenizer.from_pretrained()` in the loader and the `isinstance(t5_tokenizer, T5Tokenizer)` assertion in the text encoder invocation.
3c	FLUX with long prompt (truncation path)	Use a very long prompt (500+ words) with a FLUX model. Check that: the image generates, and the console shows a truncation warning (this exercises the tokenizer's truncation detection logic).
3d	SD3 text-to-image	Load an SD3 model. Generate an image with a simple prompt. This covers the SD3 text encoder's T5 tokenization path and its `batch_decode` for truncation warnings.
3e	Model size calculation	Load a FLUX or SD3 model and check that the model manager correctly reports the tokenizer memory footprint in the logs (exercises the `isinstance(model, T5Tokenizer)` path in `model_util.py`). No crash = pass.

Change 4: `configure_http_backend` removed + session-aware metadata fetching

Files:

invokeai/backend/model_manager/metadata/metadata_base.py — removed configure_http_backend import and call
invokeai/backend/model_manager/metadata/fetch/huggingface.py — removed configure_http_backend, added _model_info_via_session() fallback, added _has_custom_session flag

Why (root cause chain):

transformers>=5.1.0 pulls in huggingface_hub>=1.0.0 as a dependency.
huggingface_hub 1.0 switched its HTTP backend from requests to httpx and removed the configure_http_backend() function entirely.
InvokeAI called configure_http_backend(backend_factory=lambda: session) in two places to inject a custom requests.Session — this was used in production for download_urls() and, critically, in tests to inject a TestSession with mock HTTP adapters so tests could run without real network calls.
Simply removing the calls fixed the import crash in production (since HfApi() now uses httpx internally and works fine for real HTTP). However, it broke the test suite: HfApi().model_info() now bypasses the mock requests.TestSession entirely and hits the real HuggingFace API, causing RepositoryNotFoundError for test-only repos like InvokeAI-test/textual_inversion_tests.
The fix: HuggingFaceMetadataFetch now tracks whether a custom session was injected (_has_custom_session). When true, from_id() calls a new _model_info_via_session() method that uses the injected requests.Session to query the HF API directly (matching the URL patterns the test mocks expect). When false (production), it uses HfApi() as before.

#	Test	How
4a	Import smoke test	`python -c "from invokeai.backend.model_manager.metadata.fetch import HuggingFaceMetadataFetch"`
4b	Automated test suite	`pytest tests/app/services/model_install/test_model_install.py -x -v` — all 19 tests should pass, especially `test_heuristic_import_with_type`, `test_huggingface_install`, and `test_huggingface_repo_id` which depend on mock HF API responses via the injected session.
4c	Install a model from HuggingFace	In the UI's Model Manager, add a model by HuggingFace repo ID (e.g. `stabilityai/sd-turbo`). Confirm the metadata (name, description, tags) is correctly fetched and displayed, and the model downloads successfully. This exercises the production `HfApi().model_info()` path, plus `hf_hub_url()` and `download_urls()`.
4d	Browse HF model metadata	If the UI has a model info/details view, open it for an already-installed HF model and confirm metadata fields are populated.

Change 5: `HfFolder` → `huggingface_hub.get_token()`

File: invokeai/app/services/model_install/model_install_default.py
Why: HfFolder was removed in huggingface_hub 1.0+. The replacement is the top-level get_token() function.

#	Test	How
5a	Import smoke test	`python -c "from invokeai.app.services.model_install.model_install_default import ModelInstallService"`
5b	Install gated model (with token)	If you have a HuggingFace account with an access token cached (`huggingface-cli login`), try installing a gated model (e.g. `black-forest-labs/FLUX.1-dev`). Confirm the token is automatically injected and the download succeeds.
5c	Install public model (no token)	Without explicit token, install a public model. Confirm `get_token()` returns `None` gracefully and the install proceeds.

Change 6: `transformers>=5.1.0` override for compel

File: pyproject.toml — override-dependencies
Why: compel==2.1.1 requires transformers ~= 4.25 (<5.0). The uv override forces past this constraint.

#	Test	How
6a	SD 1.5 prompt weights	Generate an image with weighted prompts: `"a (red:1.5) car on a (blue:0.5) road"`. Compare to an unweighted `"a red car on a blue road"`. The weighted version should show noticeably more red and less blue. This is the core compel functionality.
6b	SD 1.5 negative prompts	Generate with prompt `"a photo of a dog"` and negative prompt `"blurry, low quality"`. Confirm it generates without crash.
6c	SDXL prompt weights	Same as 6a but with an SDXL model. SDXL uses a different compel path (`SDXLCompelPromptInvocation`).
6d	Prompt blending (compel syntax)	Try compel blend syntax if supported: `"a photo of a cat".blend("a photo of a dog", 0.5)` or `("a cat", "a dog").blend(0.5, 0.5)`. This exercises deeper compel internals.

Change 7: T5 shared-weight assertion → `model.tie_weights()`

Files:

invokeai/backend/model_manager/load/model_loaders/flux.py — _load_state_dict_into_t5() classmethod
invokeai/backend/quantization/scripts/quantize_t5_xxl_bnb_llm_int8.py — load_state_dict_into_t5() function

Why (root cause chain):

T5 models have a shared weight: model.shared.weight and model.encoder.embed_tokens.weight should refer to the same tensor.
In transformers 4.x, this sharing was implemented as a Python object alias — both attributes literally pointed to the same nn.Parameter object, so a is b was True.
In transformers 5.x, weight tying is implemented at the parameter level via _tie_weights() / tie_weights(). The two attributes may be distinct nn.Parameter objects that are kept in sync by the framework, so a is b can be False.
InvokeAI calls model.load_state_dict(state_dict, strict=False, assign=True). The assign=True flag replaces parameters in-place rather than copying data into existing tensors. This severs even the parameter-level tie that transformers 5.x establishes.
The old code then asserted model.encoder.embed_tokens.weight is model.shared.weight, which was guaranteed True in 4.x but fails in 5.x after assign=True.
Fix: Replace the identity assertion with model.tie_weights(), which re-establishes the tie regardless of how it is internally implemented. This is forward-compatible and is the officially recommended approach.

#	Test	How
7a	Import smoke test	`python -c "from invokeai.backend.model_manager.load.model_loaders.flux import FluxBnbQuantizednf4bCheckpointModel"`
7b	FLUX text-to-image	Load a FLUX model and generate an image. This is the primary code path that calls `_load_state_dict_into_t5()`. The generation should complete without `AssertionError`. (Same as test 3b — this change and Change 3 are both exercised together.)
7c	FLUX BnB quantized model	If you have a BnB-quantized FLUX model, load and generate with it. This exercises the `FluxBnbQuantizednf4bCheckpointModel` loader which also calls `_load_state_dict_into_t5()`.
7d	Quantize script (manual)	If you need to re-quantize a T5 model: run `quantize_t5_xxl_bnb_llm_int8.py` and confirm it completes without assertion errors.

Change 8: `HFTokenHelper.get_status()` — null token guard for `get_token_permission()`

File: invokeai/app/api/routers/model_manager.py
Why (root cause chain):

HFTokenHelper.get_status() calls huggingface_hub.get_token_permission(huggingface_hub.get_token()) to check whether a valid HF token is present.
When no token is configured, get_token() returns None.
In huggingface_hub <1.0, get_token_permission(None) returned a falsy value, so the code fell through to return HFTokenStatus.INVALID — correct behavior.
In huggingface_hub 1.0+, get_token_permission(None) raises an exception (it now validates the input and rejects None).
The except Exception catch returned HFTokenStatus.UNKNOWN, which the frontend interprets as a network error, showing the misleading message: "Unable to Verify HF Token — Unable to verify HuggingFace token. This is likely due to a network error."
Fix: Check get_token() for None first and return INVALID immediately, before ever calling get_token_permission(). This restores the correct "no token" UI message.

#	Test	How
8a	No token → INVALID status	Remove/rename your HF token file (`~/.cache/huggingface/token`), clear `$env:HF_TOKEN`, restart InvokeAI. The UI should show the proper "no token" message, not the "unable to verify / network error" message.
8b	Valid token → VALID status	Restore your token (`huggingface-cli login`), restart InvokeAI. The UI should show the token as valid.
8c	Install gated model without token	With no token, try to install a gated model (e.g. `black-forest-labs/FLUX.1-dev`). The UI should clearly indicate a token is needed, not a network error.

Automated Test Suite

#	Command	What it covers
A1	`pytest ./tests -x -m "not slow"`	Run the full fast test suite. Any existing tests that touch model loading, metadata, or imports will catch regressions. The `-x` flag stops on first failure for quick feedback.
A2	`pytest ./tests -x -m "slow"`	Run slow tests (if you have models available). These likely include integration tests.

Quick Smoke Test Script (all imports at once)

Run this to verify none of the changed files crash on import:

python -c "
from invokeai.backend.stable_diffusion.diffusers_pipeline import StableDiffusionGeneratorPipeline
from invokeai.backend.image_util.safety_checker import SafetyChecker
from invokeai.app.invocations.flux_text_encoder import FluxTextEncoderInvocation
from invokeai.app.invocations.sd3_text_encoder import Sd3TextEncoderInvocation
from invokeai.backend.model_manager.load.model_util import calc_model_size_by_data
from invokeai.backend.model_manager.load.model_loaders.flux import FluxBnbQuantizednf4bCheckpointModel
from invokeai.backend.model_manager.metadata.metadata_base import HuggingFaceMetadata
from invokeai.backend.model_manager.metadata.fetch import HuggingFaceMetadataFetch
from invokeai.app.services.model_install.model_install_default import ModelInstallService
print('All imports OK')
"

If this prints All imports OK, you've passed the baseline. Then proceed to the UI-based tests in priority order: 6a → 3b/7b → 3d → 4b → 5c → 2b → 1b.

4pointoh · 2026-02-18T09:48:40Z

After further review of the compel repo, I do not feel comfortable merging this until they can add official support for Transformers 5+

I see several changes necessary in their code to fully support 5.1, and forcibly overriding the dependency to 5.1 as I've done in this PR may cause any number of unknown issues. While I did evaluate that prompt weighting itself functioned correctly and without errors (this is what Compel is used for), there is unknown risk to forcibly upgrading it to Transformers 5.1

Recommend not merging this PR until they can formally support Transformers 5+. I have opened an issue on their repo and will monitor it periodically to see if support is added.

4pointoh · 2026-02-18T09:50:58Z

 # This prevents conflicts with opencv-contrib-python, which Invoke requires.
-override-dependencies = ["opencv-python; sys_platform=='never'"]
+# Force transformers>=5.1.0 past compel==2.1.1's ~=4.25 (<5.0) constraint.
+override-dependencies = ["opencv-python; sys_platform=='never'", "transformers>=5.1.0"]


The path forward will be to undo this change once Compel formally supports Transformers 5+

Until then, while I did not face any issues throghout testing this, I do not recommend merging this PR

4pointoh · 2026-03-01T18:09:13Z

    def get_status(cls) -> HFTokenStatus:
        try:
-            if huggingface_hub.get_token_permission(huggingface_hub.get_token()):
+            token = huggingface_hub.get_token()


Why was this changed? Answer: During testing I found that the new version of HF Hub now throws an exception if you pass a None token to get_token_permissions. This did not happen in the old version. As a result of this, Invoke would display the unhelpful 'An unknown error has occurred when retrieving the model from Huggingface' message rather than the very helpful 'You probably need to request permission to access this model' message.

After this change, the correct messages display as expected and the user is prompted to verify their access to the model.

4pointoh · 2026-03-01T18:10:39Z


 import torch
-from transformers import CLIPTextModel, CLIPTokenizer, T5EncoderModel, T5Tokenizer, T5TokenizerFast
+from transformers import CLIPTextModel, CLIPTokenizer, T5EncoderModel, T5Tokenizer


Why was this changed? Answer: T5TokenizerFast has been removed in Transformers 5.

4pointoh · 2026-03-01T18:10:49Z

            context.util.signal_progress("Running T5 encoder")
            assert isinstance(t5_text_encoder, T5EncoderModel)
-            assert isinstance(t5_tokenizer, (T5Tokenizer, T5TokenizerFast))
+            assert isinstance(t5_tokenizer, T5Tokenizer)


Why was this changed? Answer: T5TokenizerFast has been removed in Transformers 5.

4pointoh · 2026-03-01T18:11:46Z

        # Add user's cached access token to HuggingFace requests
        if source.access_token is None:
-            source.access_token = HfFolder.get_token()
+            source.access_token = hf_get_token()


Why was this changed? Answer: Updating transformers also mandated updating the huggingface_hub dependency. This is the new way to access the get_token function.

4pointoh · 2026-03-01T18:12:15Z

 from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker
 from PIL import Image, ImageFilter
-from transformers import AutoFeatureExtractor
+from transformers import AutoImageProcessor


Why was this changed? Answer: AutoFeatureExtractor has been replaced with AutoImageProcesser in Transformers 5

4pointoh · 2026-03-01T18:12:52Z

+                cls.feature_extractor.save_pretrained(model_path)
                cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(repo_id)
-                cls.safety_checker.save_pretrained(model_path, safe_serialization=True)
+                cls.safety_checker.save_pretrained(model_path)


Why was this changed? Answer: safe_serialization is no longer an optional parameter in Transformers 5, it is always enabled.

4pointoh · 2026-03-01T18:13:03Z

    CLIPTokenizer,
    T5EncoderModel,
-    T5TokenizerFast,
+    T5Tokenizer,


Why was this changed? Answer: T5TokenizerFast has been removed in Transformers 5.

4pointoh · 2026-03-01T18:13:20Z

+        # parameter level (via _tie_weights / tie_weights) rather than as a Python object
+        # alias.  load_state_dict(assign=True) replaces parameters in-place, which severs
+        # the parameter-level tie.  Calling tie_weights() re-establishes it.
+        model.tie_weights()


Why was this changed? Answer: See comment

4pointoh · 2026-03-01T18:13:29Z

 from diffusers.pipelines.pipeline_utils import DiffusionPipeline
 from diffusers.schedulers.scheduling_utils import SchedulerMixin
-from transformers import CLIPTokenizer, PreTrainedTokenizerBase, T5Tokenizer, T5TokenizerFast
+from transformers import CLIPTokenizer, PreTrainedTokenizerBase, T5Tokenizer


Why was this changed? Answer: T5TokenizerFast has been removed in Transformers 5.

4pointoh · 2026-03-01T18:15:38Z

        """
        self._requests = session or requests.Session()
-        configure_http_backend(backend_factory=lambda: self._requests)
+        self._has_custom_session = session is not None


Why was this changed? Answer: Invoke's unit test suite depended on the function configure_http_backend to "mock" callouts inside the test. This is how Invoke ensures that we are not making real callouts to real services inside of the test.

configure_http_backend was removed from the huggingface_hub package entirely. So, we need to re-implement this functionality a different way.

4pointoh · 2026-03-01T18:17:34Z

        metadata = HuggingFaceMetadata.model_validate_json(json)
        return metadata

+    def _model_info_via_session(self, repo_id: str, variant: Optional[ModelRepoVariant] = None) -> SimpleNamespace:


This is how we implement the functionality which mocks the session. This code only ever executes in unit tests, and only executes when attempting to make a callout to the real internet. We return this mocked response instead. This mimics what we were already doing with configure_http_backend

4pointoh · 2026-03-01T18:18:49Z

        full-precision model is returned.
        """
        session = session or Session()
-        configure_http_backend(backend_factory=lambda: session)  # used in testing


Why was this changed? Answer: Since we've implement the mocking directly in the hugginface module, it no longer needs to be manually mocked elsewhere.

4pointoh · 2026-03-01T18:19:02Z

+    # parameter level (via _tie_weights / tie_weights) rather than as a Python object
+    # alias.  load_state_dict(assign=True) replaces parameters in-place, which severs
+    # the parameter-level tie.  Calling tie_weights() re-establishes it.
+    model.tie_weights()


Why was this changed? Answer: See comment

4pointoh · 2026-03-01T18:19:30Z

 from diffusers.utils.import_utils import is_xformers_available
 from pydantic import Field
-from transformers import CLIPFeatureExtractor, CLIPTextModel, CLIPTokenizer
+from transformers import CLIPImageProcessor, CLIPTextModel, CLIPTokenizer


Why was this changed? Answer: CLIPFeatureExtractor has been replaced with ClipImageProcessor in Transformers 5

lstein · 2026-05-11T04:43:00Z

Thanks for your patience and for all the work that went into this. The next release will focus on updating Transformers and other out of date libraries. A couple of questions:

What is the status of the Compel update? It looks like there's a PR there that is ready to go.
There are conflicts with the main branch. Could you resolve them as soon as convenient?

kappacommit · 2026-05-11T06:19:49Z

Thanks for your patience and for all the work that went into this. The next release will focus on updating Transformers and other out of date libraries. A couple of questions:
1. What is the status of the Compel update? It looks like there's a PR there that is ready to go.

2. There are conflicts with the main branch. Could you resolve them as soon as convenient?

There is a PR but it has sat for a while. It looks like it has had some movement recently though. Lets hope it gets merged damian0815/compel#129

Regarding the merge conflicts, this PR is now severely out of date, in particular because of all of the new models added in 6.13, these all need to be updated.

I will update this once we have a path forward on Compel. Until then, it is a moving target, as we're likely to keep adding dependencies on the diffusers 4 concepts until we can cut over to diffusers 5.

lstein · 2026-05-29T02:51:20Z

Hi,

I've been unable to reach Damian regarding progress towards updating compel to use Transformers 5.1.0. So for the time being I have forked his repository into the invoke-ai organization and applied the 5.1.0 patch located at damian0815/compel#129.

The enclosed patch file will update pyproject.toml to use our fork of compel and updates uv.lock as well. None of this has been tested.

8859.compel.patch

kappacommit · 2026-05-29T09:24:57Z

New PR open here with compel patch applied: #9248

Closing this one

* Update to Transformers 5.1.0 * remove extra stuff * feat: ERNIE-Image integration (incl. diffusers 0.38 + transformers 5) Adds Baidu ERNIE-Image and ERNIE-Image-Turbo as a new BaseModelType, mirroring the FLUX.2 / Z-Image integration pattern. Both models share the ErnieImageTransformer2DModel architecture (3072 hidden, 24 layers, 24 heads) and an AutoencoderKLFlux2 VAE; they differ only in default inference settings (50 steps + CFG 4.0 vs 8 steps + CFG 1.0 for Turbo). Built on top of PR #8859 (transformers 5.1+) and additionally bumps diffusers 0.36.0 -> 0.38.0, which is the first release containing the ErnieImagePipeline and ErnieImageTransformer2DModel. Backend - BaseModelType.ErnieImage, ModelType.PromptEnhancer, two new SubModelTypes (pe, pe_tokenizer) for the bundled prompt enhancer - Main_Diffusers_ErnieImage_Config + Main_Checkpoint_ErnieImage_Config with state-dict-based detection (x_embedder + text_proj + adaLN_modulation) - Diffusers loader registered for ERNIE-Image; uses upstream subdir conventions, loads transformer / vae / text_encoder / tokenizer plus optional pe / pe_tokenizer - New invokeai/backend/ernie_image/ with sampling utilities (2x2 patchify, BN normalize/denormalize, sigma schedule, padded text packing) and a rectified-flow denoise loop supporting Euler/Heun/LCM - Five invocations: model_loader, text_encoder (with prompt-enhancer toggle), denoise, vae_encode, vae_decode - ErnieImageConditioningInfo + ConditioningField/Output + pickle allowlist - ERNIE_IMAGE_SCHEDULER_MAP reusing the FlowMatch* scheduler classes - New generation modes ernie_image_{txt2img,img2img,inpaint,outpaint} - Starter models for baidu/ERNIE-Image and baidu/ERNIE-Image-Turbo + STARTER_BUNDLES entry Frontend - Regenerated services/api/schema.ts to expose the new node types - Type unions extended (ImageOutput / LatentToImage / ImageToLatents / DenoiseLatents / MainModelLoaderNodes) - ParamsState gains ernieImageScheduler + ernieImageUsePromptEnhancer, with reducers, selectors, and selectIsErnieImage - buildErnieImageGraph (txt2img/img2img/inpaint/outpaint) wired into useEnqueueCanvas - ERNIE entries added to MODEL_BASE_TO_{COLOR,LONG_NAME,SHORT_NAME} and prompt_enhancer to MODEL_TYPE_TO_LONG_NAME - ParamErnieImageScheduler and ParamErnieImagePromptEnhancer rendered conditionally in GenerationSettingsAccordion - All add{TextTo,ImageTo,Inpaint,Outpaint}Image type guards extended to accept ernie_image_denoise; isMainModelWithoutUnet ditto Diffusers 0.38 fix-out - hotfixes.py: import LoRACompatibleConv directly; the lazy-module __getattr__ no longer exposes diffusers.models.lora as an attribute Verification - pytest tests/ -m "not slow": 619 passed, 0 failed - pnpm lint:tsc / lint:eslint / lint:prettier: clean - pnpm test:no-watch: 563 passed, 0 failed - Manual smoketest pending: requires baidu/ERNIE-Image weights and a GPU (8B parameters; CPU not practical) Out of scope (follow-up phases) - ControlNet, IP-Adapter, LoRA support for ERNIE-Image - Single-file checkpoint loading (defensive scaffolding only) - Metadata recall handlers in the gallery side panel * Chore Ruff + Typegen * Chore ruff * fix(ernie-image): timestep scale, live preview, missing graph case, and UI cleanup - Pass timesteps in [0, num_train_timesteps] to the transformer instead of [0, 1]; the diffusers Timesteps embedding expects the unnormalised range, which produced mosaic-pattern garbage instead of an image. - Unpatchify predicted-x0 before the denoise step callback and route through sd_step_callback so the canvas shows a live preview during sampling (uses FLUX.2's RGB factors -- same AutoencoderKLFlux2 / 32 latent channels). - Add the ernie-image case to useEnqueueGenerate (Generate tab); was only wired up in useEnqueueCanvas, so plain text-to-image failed with "No graph builders for base ernie-image". - Move the Prompt Enhancer toggle from the Generation accordion into a dedicated ERNIE-Image block in the Advanced accordion; hide the rest of the SD-style advanced controls (CLIP skip, CFG rescale, seamless, color comp., separate VAE) since none apply to ERNIE-Image. - Detect ERNIE-Image-Turbo by name in MainModelDefaultSettings.from_base so installs (starter or manual) get steps=8, cfg_scale=1.0 instead of the standard 50/4.0. - Pin compel to Cstannahill/compel5@chore/transformers5-diffusers-smoke for transformers>=5 compatibility (PR damian0815/compel#129). * Merged missed * Chore Ruff * Chore Fix UV lock * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE's denoise node has no denoise_mask input, so masked modes are unsupported. Drop img2img/inpaint/outpaint from the ERNIE graph builder, exclude ernie_image_denoise from a new MaskableDenoiseNodes type used by addInpaint/addOutpaint, and align addImageToImage unions with the node type aliases. Fixes tsc failures on the ernie-image branch. * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE-Image's denoise node has no denoise_mask input and no mask logic, so masked modes (inpaint/outpaint) are impossible and image-to-image is dropped as well. Frontend: - buildErnieImageGraph now builds txt2img only; asserts on other modes - add MaskableDenoiseNodes (DenoiseLatentsNodes minus ernie_image_denoise) and use it in addInpaint/addOutpaint - drop ernie_image_vae_encode from ImageToLatentsNodes; align addImageToImage unions with the LatentToImage/ImageToLatents aliases - regenerate schema.ts Backend: - delete the ernie_image_vae_encode invocation (i2l, only used by the removed image-input modes) - drop ernie_image_{img2img,inpaint,outpaint} from GENERATION_MODES Fixes the tsc failures on the ernie-image branch. * Chore OpenApi * fix(model_manager): remove duplicate _has_anima_keys shadowing ComfyUI-bundled prefix support A duplicate _has_anima_keys definition (older, net.-only) shadowed the complete version that also recognizes the `model.diffusion_model.` ComfyUI-bundled prefix, causing Anima identification to reject bundled checkpoints. * Chore Openapi * fix(ernie-image): exclude prompt enhancer from fp8, tighten turbo detection Address review findings on #9115: - Exclude SubModelType.PromptEnhancer and PromptEnhancerTokenizer from fp8 layerwise casting. The prompt enhancer is a causal LM driven by generate() — one full forward per generated token — so casting made the whole LM round-trip bf16<->fp8 per token, on top of fp8 rounding a model whose entire job is text quality. - Match turbo detection on the install directory's leaf name instead of the whole path string. An in-place install records an absolute path, so an ancestor directory like /mnt/turbo-nvme/ silently gave the base model Turbo's 8 steps and CFG 1.0. - Pass local_files_only=True on all ERNIE-Image from_pretrained calls and load tokenizers bare, matching the sibling loaders. - Cap the prompt enhancer's max_new_tokens at 1024. Driving it off model_max_length hangs the graph if the tokenizer config omits it and transformers substitutes its int(1e30) sentinel. Adds regression tests for the fp8 exclusion and the turbo path matching. * fix(ernie-image): honor denoising window, noise init latents, harden PE gate Self-review follow-ups on #9115: - Honor denoising_end. Every FlowMatch scheduler appends its own terminal 0 sigma, and passing the window minus its last entry let that zero stand in for the requested end sigma - so denoising_end < 1.0 ran a full denoise in fewer, coarser steps. Hand the scheduler the whole window and truncate its appended zero instead, which also keeps the scheduler's own `shift` applied to the terminal sigma. - Reject a denoising window that rounds down to a single sigma. It yielded zero steps, so the loop returned its input untouched and the graph decoded raw noise with no error at all. - Blend image-to-image init latents with noise at the first sigma, and reject denoising_start > 0 when no latents are provided. Both cases previously lied to the model about where the sample sits on the rectified-flow path. The shape check now covers batch and spatial dims, and its message no longer points at a VAE-encode node that does not exist. - Require both `pe` and `pe_tokenizer` to be present AND declared in model_index.json before offering the prompt enhancer. get_hf_load_class resolves submodels from that file, so a directory-only check let a partial install pass the gate and then hard-fail the generation - by default, since the toggle is on. - Pass the real generation dimensions to the prompt enhancer instead of leaving it on its 1024x1024 defaults. Adds regression tests for each, including a graph-builder test suite for ERNIE (which had none). Added ERNIE-Image + Krea 2 Raw into README.md under Supported Model. * Readme.me update with ERNIE-Image-Turbo * fix(ernie-image): seed the stochastic scheduler, blend at the shifted sigma Round-4 review follow-ups on #9115: - Pass a seeded generator into scheduler.step. FlowMatchLCMScheduler is stochastic - it re-noises the sample every step - so with generator=None it drew from the global RNG. The seed field only controlled the initial latent, so the same seed produced a different image on every run and seed recall from gallery metadata could not reproduce an LCM generation. Seeded from `seed ^ 0xFFFFFFFF` (as denoise_latents.py does) so the step noise stays decorrelated from the initial noise, and from a CPU generator so it is device-independent like the initial noise. - Blend image-to-image init latents at the scheduler's post-shift first sigma instead of the raw schedule value. get_schedule emits raw linspace values and the scheduler applies `shift` in set_timesteps, so the blend built the sample at one sigma and then told the first model call it was at another. The blend moves into denoise(), which is the only layer that knows the shifted sigma. - Add an `add_noise` field mirroring z_image_denoise. The only node that can currently feed `latents` is another ernie_image_denoise, whose output already sits at the handoff sigma - re-noising it broke the very multi-stage handoff the denoising_end fix enables. - Give the prompt enhancer the original size rather than the intermediate scaled render size. Regenerates openapi.json / schema.ts for the new field. Adds regression tests for the seed contract (same seed identical, different seeds different, euler unaffected) and for the post-shift blend. --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>

Your Name added 2 commits February 6, 2026 19:58

Update to Transformers 5.1.0

df20012

remove extra stuff

f69f22c

4pointoh requested review from JPPhoto, blessedcoolant, dunkeroni and lstein as code owners February 7, 2026 01:14

github-actions Bot added api python PRs that change python files Root invocations PRs that change invocations backend PRs that change backend files services PRs that change app services python-deps PRs that change python dependencies labels Feb 7, 2026

Merge branch 'main' into main

57e91b4

lstein assigned Pfannkuchensack and blessedcoolant Feb 16, 2026

4pointoh commented Feb 18, 2026

View reviewed changes

4pointoh commented Mar 1, 2026

View reviewed changes

lstein added this to Invoke - Community Roadmap Mar 7, 2026

lstein moved this to 6.14.x in Invoke - Community Roadmap Mar 7, 2026

lstein added the v6.14.x label Mar 7, 2026

lstein moved this from 6.14.x Theme: LIBRARY UPDATES to 6.13.x Theme: MODELS in Invoke - Community Roadmap Mar 24, 2026

lstein moved this from 6.13.x Theme: MODELS to 6.14.x Theme: LIBRARY UPDATES in Invoke - Community Roadmap Mar 24, 2026

Pfannkuchensack mentioned this pull request May 3, 2026

feat: ernie image/turbo #9115

Merged

5 tasks

lstein added 6.14.0 and removed v6.14.x labels May 9, 2026

lstein moved this from 6.14.x Theme: LIBRARY UPDATES to 6.13.5 LIBRARY UPDATES in Invoke - Community Roadmap May 12, 2026

lstein added 6.13.5 Library Updates and removed 6.14.0 labels May 12, 2026

4pointoh closed this May 29, 2026

Uh oh!

Conversation

4pointoh commented Feb 7, 2026

Summary

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

Change Description & Test Plan (Generated with AI Assistance)

Regression Testing Plan: Transformers 5.1.0 + HuggingFace Hub Migration

Change 1: CLIPFeatureExtractor → CLIPImageProcessor

Change 2: AutoFeatureExtractor → AutoImageProcessor + removed safe_serialization

Change 3: T5TokenizerFast → T5Tokenizer (4 files)

Change 4: configure_http_backend removed + session-aware metadata fetching

Change 5: HfFolder → huggingface_hub.get_token()

Change 6: transformers>=5.1.0 override for compel

Change 7: T5 shared-weight assertion → model.tie_weights()

Change 8: HFTokenHelper.get_status() — null token guard for get_token_permission()

Automated Test Suite

Quick Smoke Test Script (all imports at once)

Uh oh!

4pointoh commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

4pointoh Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lstein commented May 11, 2026

Uh oh!

kappacommit commented May 11, 2026

Uh oh!

lstein commented May 29, 2026

Uh oh!

kappacommit commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Change 1: `CLIPFeatureExtractor` → `CLIPImageProcessor`

Change 2: `AutoFeatureExtractor` → `AutoImageProcessor` + removed `safe_serialization`

Change 3: `T5TokenizerFast` → `T5Tokenizer` (4 files)

Change 4: `configure_http_backend` removed + session-aware metadata fetching

Change 5: `HfFolder` → `huggingface_hub.get_token()`

Change 6: `transformers>=5.1.0` override for compel

Change 7: T5 shared-weight assertion → `model.tie_weights()`

Change 8: `HFTokenHelper.get_status()` — null token guard for `get_token_permission()`

4pointoh Mar 1, 2026 •

edited

Loading