Feat(model support): ideogram4 support

README.md

@@ -75,6 +75,7 @@ Invoke features an organized gallery system for easily storing, accessing, and r
 - Anima
 - Qwen Image
 - Qwen Image Edit
+- Ideogram 4
 - Nano Banana (API Only)
 - GPT Image (API Only)
 - Wan (API Only)

docs/src/content/docs/concepts/models.mdx

@@ -51,6 +51,44 @@ In this situation, you may need to provide some additional information to identi
 
 [set up in the config file]: ../../configuration/invokeai-yaml
 
+## Ideogram 4
+
+Ideogram 4 is an open-weight text-to-image model with a distinctive **structured JSON prompt**: instead of a single sentence, the model is trained to read an overall scene description plus a list of regions, each with a bounding box and its own text. Invoke assembles this JSON for you.
+
+### Installing Ideogram 4
+
+The weights are gated on HuggingFace under a **non-commercial license**. Open the model page, accept the terms, and make sure your HuggingFace token is [set up in the config file] before installing. Two builds are available:
+
+- [`ideogram-ai/ideogram-4-nf4`](https://huggingface.co/ideogram-ai/ideogram-4-nf4) — nf4 quantized, **CUDA only**, fits in 24 GB VRAM. Recommended for NVIDIA GPUs.
+- [`ideogram-ai/ideogram-4-fp8`](https://huggingface.co/ideogram-ai/ideogram-4-fp8) — fp8 quantized, runs on **any device**, with higher memory use.
+
+Paste either repo ID into the Model Manager's HuggingFace / URL field to install.
+
+### Prompting Ideogram 4
+
+When an Ideogram 4 model is selected, Invoke builds the structured JSON prompt automatically:
+
+- The **positive prompt** becomes the overall scene description.
+- Each enabled **Regional Guidance** layer on the Canvas contributes one element: its drawn box becomes the region's bounding box and its prompt becomes that region's description. Draw a box where you want something and describe it there.
+- To drive the model directly, paste a **raw JSON** object into the prompt box — anything starting with `{` is passed through unchanged.
+
+The exact JSON that was encoded is stored in the image metadata as **Structured Caption**, and can be recalled straight back into the prompt box from the metadata viewer.
+
+:::note[No negative prompt]
+Ideogram 4 does not use a negative prompt — it has a dedicated unconditional branch instead, so the negative prompt box has no effect.
+:::
+
+### Generation settings
+
+- **Sampler Preset** — the primary quality/speed control. `Quality (48 steps)`, `Default (20 steps)`, and `Turbo (12 steps)` each bundle a step count, a guidance schedule, and the schedule shift.
+- **Advanced** overrides (all optional, leave on *Auto* to use the preset's values): **Steps**, **Guidance Scale**, **Schedule Shift (mu)**, and a **Color Palette** that biases the generated colors.
+
+:::caution[Built-in safety filter]
+Ideogram 4 ships with a **content safety filter baked into the released model weights**. When it triggers, the model outputs a placeholder image reading *"Image blocked by safety filter"* instead of your image. This is the model's own filter — it is **not** Invoke's NSFW checker (it fires even with the NSFW checker off) and cannot be disabled from Invoke.
+
+The filter is known to **false-positive on completely benign prompts**, and the quantized builds can trigger it more readily. Invoke always sends the model a structured JSON prompt, which tends to trip the filter less than plain text; if you hit it, rephrasing or adjusting the prompt usually clears it.
+:::
+
 ## Editing model metadata
 
 Every model has an editable **Source URL** field alongside its name and description. Use it to record where a model came from — for example a Civitai or HuggingFace page — independent of how it was originally installed. The URL is editable from the model's **Edit** view and appears as a clickable link in the model header once set. Models without a URL simply hide the field.

invokeai/app/api/dependencies.py

@@ -60,6 +60,7 @@
     CogView4ConditioningInfo,
     ConditioningFieldData,
     FLUXConditioningInfo,
+    Ideogram4ConditioningInfo,
     QwenImageConditioningInfo,
     SD3ConditioningInfo,
     SDXLConditioningInfo,
@@ -152,6 +153,7 @@ def initialize(
                     SD3ConditioningInfo,
                     CogView4ConditioningInfo,
                     ZImageConditioningInfo,
+                    Ideogram4ConditioningInfo,
                     QwenImageConditioningInfo,
                     AnimaConditioningInfo,
                 ],

invokeai/app/invocations/fields.py

@@ -344,6 +344,12 @@ class ZImageConditioningField(BaseModel):
     )
 
 
+class Ideogram4ConditioningField(BaseModel):
+    """An Ideogram 4 conditioning tensor primitive value"""
+
+    conditioning_name: str = Field(description="The name of conditioning tensor")
+
+
 class QwenImageConditioningField(BaseModel):
     """A Qwen Image Edit conditioning tensor primitive value"""
 

invokeai/app/invocations/ideogram4_caption.py

@@ -0,0 +1,91 @@
+from typing import Annotated, Optional
+
+from pydantic import BaseModel, Field, field_validator
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
+from invokeai.app.invocations.fields import InputField, UIComponent
+from invokeai.app.invocations.primitives import StringOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.ideogram4.caption import build_ideogram4_caption
+
+_IDEOGRAM4_COORD_MAX = 1000
+
+# Exactly four normalized coordinates [y_min, x_min, y_max, x_max], each in 0..1000. Modeled as a
+# constrained type so the generated OpenAPI schema advertises minItems/maxItems and per-item
+# minimum/maximum — clients get the contract from the schema, not only from runtime validation.
+Ideogram4Bbox = Annotated[
+    list[Annotated[int, Field(ge=0, le=_IDEOGRAM4_COORD_MAX)]],
+    Field(min_length=4, max_length=4),
+]
+
+
+class Ideogram4Region(BaseModel):
+    """A single region of an Ideogram 4 structured caption (description + optional bounding box)."""
+
+    prompt: str = Field(description="The region's description (becomes the element's `desc`).")
+    bbox: Optional[Ideogram4Bbox] = Field(
+        default=None,
+        description="Normalized bounding box [y_min, x_min, y_max, x_max] (0–1000), or null for a region "
+        "with no drawn content.",
+    )
+
+    @field_validator("bbox")
+    @classmethod
+    def _validate_bbox(cls, v: Optional[list[int]]) -> Optional[list[int]]:
+        """Enforce the ordering the constrained type can't express: y_min <= y_max and x_min <= x_max.
+
+        Length (exactly 4) and range (0..1000) are enforced by the Ideogram4Bbox type. The caption
+        builder forwards the bbox verbatim into the structured JSON, so an inverted box would emit a
+        malformed prompt the model may misapply — reject it here.
+        """
+        if v is None:
+            return v
+        y_min, x_min, y_max, x_max = v
+        if y_min > y_max or x_min > x_max:
+            raise ValueError(
+                f"bbox must satisfy y_min <= y_max and x_min <= x_max, got [y_min={y_min}, x_min={x_min}, "
+                f"y_max={y_max}, x_max={x_max}]"
+            )
+        return v
+
+
+@invocation(
+    "ideogram4_caption_builder",
+    title="Caption Builder - Ideogram 4",
+    tags=["prompt", "ideogram4"],
+    category="conditioning",
+    version="1.0.0",
+    classification=Classification.Prototype,
+)
+class Ideogram4CaptionBuilderInvocation(BaseInvocation):
+    """Assembles the Ideogram 4 structured JSON caption at generation time.
+
+    The caption is built here (not in the graph builder) so the batch-injectable global `prompt` — which
+    dynamic prompts and prompt batching vary — is folded into the encoded caption. The regions and color
+    palette are fixed per generation and supplied as inputs. If the prompt is already a JSON object it is
+    passed through verbatim; otherwise it is always wrapped in the structured JSON schema (never bare
+    plain text — Ideogram's safety filter false-positives far more on plain text).
+    """
+
+    prompt: str = InputField(
+        default="",
+        description="The global prompt (becomes `high_level_description`, or is used verbatim if it is "
+        "already a JSON caption).",
+        ui_component=UIComponent.Textarea,
+    )
+    regions: list[Ideogram4Region] = InputField(
+        default=[],
+        description="Regional descriptions and bounding boxes assembled from Canvas Regional Guidance layers.",
+    )
+    color_palette: list[str] = InputField(
+        default=[],
+        description="Optional color palette as hex colors (#RRGGBB).",
+    )
+
+    def invoke(self, context: InvocationContext) -> StringOutput:
+        caption = build_ideogram4_caption(
+            self.prompt,
+            [(region.prompt, region.bbox) for region in self.regions],
+            self.color_palette,
+        )
+        return StringOutput(value=caption)

invokeai/app/invocations/ideogram4_denoise.py

@@ -0,0 +1,179 @@
+from typing import Literal, Optional
+
+import torch
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, invocation
+from invokeai.app.invocations.fields import (
+    FieldDescriptions,
+    Ideogram4ConditioningField,
+    Input,
+    InputField,
+)
+from invokeai.app.invocations.model import TransformerField
+from invokeai.app.invocations.primitives import LatentsOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.step_callback import (
+    FLUX2_LATENT_RGB_BIAS,
+    FLUX2_LATENT_RGB_FACTORS,
+    sample_to_lowres_estimated_image,
+)
+from invokeai.backend.ideogram4 import run_ideogram4_denoise
+from invokeai.backend.ideogram4.latent_norm import get_latent_norm
+from invokeai.backend.ideogram4.sampler_configs import PRESETS
+from invokeai.backend.ideogram4.sampling_utils import unpatchify_and_denormalize
+from invokeai.backend.ideogram4.transformer_pair import Ideogram4TransformerPair
+from invokeai.backend.stable_diffusion.diffusion.conditioning_data import Ideogram4ConditioningInfo
+from invokeai.backend.util.devices import TorchDevice
+
+# Named sampler presets bundle step count, guidance schedule (with polish tail), and the
+# logit-normal schedule mean/std. V4_QUALITY_48 is the reference default.
+IDEOGRAM4_SAMPLER_PRESETS = Literal["V4_QUALITY_48", "V4_DEFAULT_20", "V4_TURBO_12"]
+
+
+def _effective_guidance_schedule(
+    base_schedule: tuple[float, ...], preset_num_steps: int, num_steps: int, guidance_scale: Optional[float]
+) -> tuple[float, ...]:
+    """Build the per-step guidance schedule for the (possibly overridden) step count.
+
+    The preset schedule is ``(polish_gw,)*N_polish + (main_gw,)*N_main`` in loop-index order
+    (index 0 = the final/polish step). A ``guidance_scale`` override replaces the main weight while
+    the preset's polish tail is preserved; a changed step count rescales the polish tail
+    proportionally (always keeping at least one polish and one main step).
+
+    ``num_steps`` must be >= 2 (enforced by the invocation's ``steps`` field) so both a polish and a
+    main step always exist — otherwise a single step would be all-polish and silently drop the
+    ``guidance_scale`` override.
+    """
+    polish_gw = base_schedule[0]
+    main_gw = float(guidance_scale) if guidance_scale is not None else float(base_schedule[-1])
+    if num_steps == preset_num_steps and guidance_scale is None:
+        return base_schedule
+    n_polish_base = sum(1 for gw in base_schedule if gw == base_schedule[0])
+    # Cap the polish tail at num_steps - 1 so at least one main step always remains and the
+    # guidance_scale override is never silently dropped.
+    polish_count = max(1, min(round(n_polish_base * num_steps / preset_num_steps), num_steps - 1))
+    main_count = num_steps - polish_count
+    return (polish_gw,) * polish_count + (main_gw,) * main_count
+
+
+@invocation(
+    "ideogram4_denoise",
+    title="Denoise - Ideogram 4",
+    tags=["image", "ideogram4"],
+    category="latents",
+    version="1.0.0",
+    classification=Classification.Prototype,
+)
+class Ideogram4DenoiseInvocation(BaseInvocation):
+    """Runs the Ideogram 4 dual-branch flow-matching denoising loop (text-to-image)."""
+
+    transformer: TransformerField = InputField(
+        description=FieldDescriptions.transformer, input=Input.Connection, title="Transformer"
+    )
+    positive_conditioning: Ideogram4ConditioningField = InputField(
+        description=FieldDescriptions.positive_cond, input=Input.Connection
+    )
+    sampler_preset: IDEOGRAM4_SAMPLER_PRESETS = InputField(
+        default="V4_QUALITY_48",
+        description="Sampler preset (steps + guidance schedule + schedule mean/std).",
+        title="Sampler Preset",
+    )
+    width: int = InputField(default=1024, multiple_of=16, description="Width of the generated image.")
+    height: int = InputField(default=1024, multiple_of=16, description="Height of the generated image.")
+    seed: int = InputField(default=0, description="Randomness seed for reproducibility.")
+    # Optional advanced overrides of the sampler preset. None = use the preset's value.
+    steps: Optional[int] = InputField(
+        default=None,
+        ge=2,
+        le=100,
+        description="Override the preset's step count (minimum 2, so a polish and a main step both "
+        "exist). Leave empty to use the preset.",
+    )
+    guidance_scale: Optional[float] = InputField(
+        default=None,
+        ge=1.0,
+        le=20.0,
+        description="Override the main guidance weight (the preset's polish tail is preserved). "
+        "Empty = use the preset.",
+    )
+    mu: Optional[float] = InputField(
+        default=None,
+        ge=-4.0,
+        le=4.0,
+        description="Override the logit-normal schedule mean (resolution-adjusted internally). Empty = use the preset.",
+    )
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        device = TorchDevice.choose_torch_device()
+        preset = PRESETS[self.sampler_preset]
+
+        # Apply optional advanced overrides on top of the preset.
+        num_steps = self.steps if self.steps is not None else preset.num_steps
+        mu = self.mu if self.mu is not None else preset.mu
+        guidance_schedule = _effective_guidance_schedule(
+            preset.guidance_schedule, preset.num_steps, num_steps, self.guidance_scale
+        )
+
+        # Load conditioning (the stacked Qwen3-VL features).
+        cond_data = context.conditioning.load(self.positive_conditioning.conditioning_name)
+        assert len(cond_data.conditionings) == 1
+        info = cond_data.conditionings[0]
+        assert isinstance(info, Ideogram4ConditioningInfo)
+        llm_features = info.prompt_embeds.to(device=device, dtype=torch.float32)
+
+        # Progress-preview setup: Ideogram uses a FLUX.2-style 32-channel VAE, so the FLUX.2
+        # latent->RGB factors give a usable (approximate) low-res preview of the forming image at each
+        # step, without a full VAE decode. Denormalization params come from get_latent_norm (no VAE).
+        latent_shift, latent_scale = get_latent_norm()
+        rgb_factors = torch.tensor(FLUX2_LATENT_RGB_FACTORS, dtype=torch.float32)
+        rgb_bias = torch.tensor(FLUX2_LATENT_RGB_BIAS, dtype=torch.float32)
+
+        def step_callback(step: int, total: int, packed_latents: torch.Tensor) -> None:
+            preview = None
+            try:
+                # packed_latents: (1, LATENT_DIM, grid_h, grid_w) -> VAE latent (1, 32, H/8, W/8).
+                vae_latent = unpatchify_and_denormalize(
+                    packed_latents.float(),
+                    latent_shift.to(packed_latents.device),
+                    latent_scale.to(packed_latents.device),
+                )
+                preview = sample_to_lowres_estimated_image(
+                    samples=vae_latent,
+                    latent_rgb_factors=rgb_factors.to(vae_latent.device),
+                    latent_rgb_bias=rgb_bias.to(vae_latent.device),
+                )
+            except Exception:
+                # A preview must never break generation — fall back to a plain progress signal.
+                preview = None
+            if preview is not None:
+                context.util.signal_progress(
+                    "Running Ideogram 4 denoising",
+                    step / total,
+                    preview,
+                    (preview.width * 8, preview.height * 8),
+                )
+            else:
+                context.util.signal_progress("Running Ideogram 4 denoising", step / total)
+
+        transformer_info = context.models.load(self.transformer.transformer)
+        with transformer_info.model_on_device() as (_, transformers):
+            assert isinstance(transformers, Ideogram4TransformerPair)
+            packed = run_ideogram4_denoise(
+                conditional_transformer=transformers.conditional,
+                unconditional_transformer=transformers.unconditional,
+                llm_features=llm_features,
+                height=self.height,
+                width=self.width,
+                num_steps=num_steps,
+                mu=mu,
+                std=preset.std,
+                guidance_schedule=guidance_schedule,
+                seed=self.seed,
+                device=device,
+                step_callback=step_callback,
+            )
+
+        packed = packed.detach().to("cpu")
+        name = context.tensors.save(tensor=packed)
+        return LatentsOutput.build(latents_name=name, latents=packed, seed=None)