Skip to content

Krea 2 regional prompting - #9407

Merged
lstein merged 5 commits into
invoke-ai:mainfrom
JPPhoto:krea-2-regional-prompting
Jul 31, 2026
Merged

Krea 2 regional prompting#9407
lstein merged 5 commits into
invoke-ai:mainfrom
JPPhoto:krea-2-regional-prompting

Conversation

@JPPhoto

@JPPhoto JPPhoto commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds regional prompting support for Krea-2 using Flux-style restricted attention.

This PR:

  • Allows masked Krea-2 conditionings and conditioning collections.
  • Adds positive regional prompting to the canvas.
  • Applies enabled conditioning enhancers independently to global and regional prompts.
  • Accounts for retained attention masks and mask-construction scratch memory.
  • Adds CUDA-gated coverage for dense masks with memory-efficient SDPA.
  • Documents dense-mask memory costs and math-attention fallback behavior.

Related Issues / Discussions

Builds on #9406.

QA Instructions

For manual canvas QA:

  1. Select a Krea-2 model.
  2. Add a positive regional prompt.
  3. Enable Conditioning Rebalance and Seed Variance.
  4. Generate and confirm both global and regional conditionings pass through separate enhancer chains.

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions Bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs labels Jul 30, 2026
@lstein lstein added the 6.14.0 label Jul 31, 2026
@lstein lstein moved this to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Jul 31, 2026
@JPPhoto
JPPhoto force-pushed the krea-2-regional-prompting branch from e154b1c to ae77d29 Compare July 31, 2026 00:56
@Pfannkuchensack

Copy link
Copy Markdown
Member

Findings

  • Medium: invokeai/app/invocations/krea2_denoise.py:521 (_regional_attention_mask_bytes) under-reserves working memory because it ignores the float attention bias PyTorch materializes from the boolean mask on every SDPA call.
    Chain: invokeai/backend/krea2/regional_prompting.py:90 allocates the mask as torch.bool, and attention_mask_numel (invokeai/backend/krea2/regional_prompting.py:51-56) reports total_seq_len**2, i.e. exactly the bool byte count. That value plus attention_mask_build_scratch_numel is the whole of the regional contribution added to _estimate_working_memory at invokeai/app/invocations/krea2_denoise.py:566. But the mask is handed to F.scaled_dot_product_attention(..., attn_mask=attention_mask) at invokeai/backend/krea2/attention.py:87; PyTorch converts a boolean attn_mask into an additive bias in the query dtype before dispatch, so each masked attention transiently allocates a further total_seq_len**2 * 2 bytes (bf16) that nothing reserves.
    Evidence (measured on RTX 4090, torch 2.7.1+cu128, 48 heads / head_dim 128 / bf16, seq 4608): unmasked SDPA peak 54 MiB, masked peak 94 MiB, bool mask itself 20 MiB. The 40 MiB delta is exactly 4608**2 * 2. At 2560x1440 with ~400 text tokens (total_seq_len ~14800) that is ~440 MB unaccounted, on top of the ~219 MB the estimate does count per mask.
    Trigger: any regional generation at high resolution on a card where the cache places the transformer using the returned hint. The estimate carries several GB of deliberate slack (invokeai/app/invocations/krea2_denoise.py:549-554), which is why this is Medium and not High, but the PR's stated contribution is exact mask accounting and this is the largest mask-derived allocation.
    To expose this issue, add a test that asserts the regional byte estimate scales with the inference dtype width (that it includes total_seq_len**2 * dtype_bytes for the live mask in addition to the bool storage), or a CUDA-gated test that compares torch.cuda.max_memory_allocated() for one masked Krea-2 attention against _regional_attention_mask_bytes.

  • Low: invokeai/backend/krea2/regional_prompting.py:92-104 silently drops the global/background prompt when the region masks cover every grid cell, and no validator warns about it.
    Chain: background_mask starts all-True and is ANDed with ~region for each masked conditioning (line 95). If the union of regions covers all image_seq_len cells it becomes all-False. For an unmasked (global) conditioning the code then writes that all-False vector into both the text-to-image block (line 103) and the image-to-text block (line 104), so no image token attends to the global prompt and the global prompt attends to no image token; only its own diagonal text block survives. Confirmed by direct probe: with one unmasked conditioning plus an all-ones region mask, mask[global_range, image_block].any() and mask[image_block, global_range].any() are both False.
    Trigger: on the canvas, invokeai/frontend/web/src/features/nodes/util/graph/generation/buildKrea2Graph.ts always wires the global pos_prompt encoder into pos_cond_collect, so a user who draws a regional guidance layer over the whole canvas (or, at low resolution, a layer that covers every 16 px grid cell) gets their main prompt entirely ignored with no feedback. getRegionalGuidanceWarnings in invokeai/frontend/web/src/features/controlLayers/store/validators.ts:104-116 does not flag it.
    This is FLUX parity (invokeai/backend/flux/extensions/regional_prompting_extension.py:127-208 behaves identically), so it is not a regression - but Krea-2 ships it alongside a docs claim that an unmasked conditioning "applies to the background not covered by any regional mask", which is vacuous in this case.
    To expose this issue, add a test that builds an extension from one unmasked conditioning plus an all-ones region mask and asserts the intended fallback (background coverage restored, or an explicit error), rather than the current silent no-op.

@JPPhoto
JPPhoto force-pushed the krea-2-regional-prompting branch from 3bbd34e to 0f3a846 Compare July 31, 2026 01:36
@JPPhoto

JPPhoto commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

@Pfannkuchensack I think everything you raised is addressed.

…canvas graph

Adds the coverage the regional prompting change was missing:

- attention: the processor's mask/sequence-length shape guard, and that a
  processor built without regional state ignores the shared mask
- regional prompting: that only even main blocks actually apply the mask
  during a real transformer forward (the existing test only checked the
  processor map), and that an unsupported mask rank is rejected
- denoise: that each CFG pass installs a mask sized for its own text
  sequence when the positive and negative prompts tokenize to different
  lengths
- canvas graph: a composed suite running the real addRegions,
  addKrea2LoRAs and regional-guidance validators, covering validator
  filtering, the IP adapter collector teardown, the LoRA-before-regions
  ordering the regional encoders depend on, and per-region enhancers

Each test was mutation-checked against a deliberately broken
implementation to confirm it fails for the right reason.

@Pfannkuchensack Pfannkuchensack left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works.

@lstein
lstein merged commit 93fd9ae into invoke-ai:main Jul 31, 2026
17 checks passed
@JPPhoto
JPPhoto deleted the krea-2-regional-prompting branch July 31, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.0 backend PRs that change backend files docs PRs that change docs frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

3 participants