Skip to content

README.md

Latest commit

 

History

History
142 lines (85 loc) · 9.61 KB

README.md

File metadata and controls

142 lines (85 loc) · 9.61 KB
Raw

Modular Diffusers in MoDiff

MoDiff integrates the experimental Diffusers Modular Pipelines APIs with its node graph. A small set of dynamic nodes can expose different model pipelines without creating a separate hardcoded node class for every model family.

Warning

Modular Diffusers APIs and compatible Hub repositories are still evolving. A visible node contract is not proof that every model/revision will load or fit on the current hardware. Custom blocks and trust_remote_code can execute repository-supplied Python; use only reviewed, revision-pinned sources and read SECURITY.md.

Concepts

  • Dynamic node contracts: node fields adapt to the selected pipeline configuration.
  • Composable workflows: model loading, prompt encoding, denoising, and decoding can remain separate or be combined into a custom block.
  • Shared components: compatible nodes can reuse components from the package-level ComponentsManager instead of loading duplicate models.
  • Hub-backed blocks: supported repositories can provide Modular Diffusers configuration/code used to construct a node interface.
  • Resource controls: loaders expose supported quantization and offload modes, subject to package, model, and hardware compatibility.

MoDiff adapts Diffusers' Mellon node-metadata helper to supply MoDiff dynamic fields and configuration names; the derivation is recorded in pipeline_schema.py and THIRD_PARTY_NOTICES.md. Upstream Diffusers remains responsible for model components and Modular Pipeline execution.

Setup

Install and validate the backend from the repository root:

./install.sh --accelerator auto
./.venv/bin/python -m modiff.preflight --json --check-port 8088 --fail-on-error
./run.sh

On Windows, use install.ps1, .venv\Scripts\python.exe, and run.ps1 as shown in the root quick start. The managed installer owns the executable Torch profile; uv sync and uv run are intentionally unsupported.

Open http://127.0.0.1:8088. Keep the server on loopback; it has no authentication or remote-code sandbox.

Optional quantization, Nunchaku, or other acceleration paths require the matching profiles described in the root README.

Start with a bundled graph

Open the workflow library in the left sidebar and expand modular_diffusers. The repository currently includes:

  • text_to_image — a separated text-to-image pipeline.
  • image_to_image — prompt plus reference-image conditioning.
  • multiple_image_edit — multiple-image editing inputs.
  • quantization — an example with an explicit quantization configuration.
  • dynamic_node — a Hub-backed dynamic block example.

Drag a graph onto the canvas, inspect its selected model and required inputs, then update the graph before running. Models are not bundled with these JSON files; MoDiff may need to download them, and gated repositories may require accepted terms plus a Hugging Face read token.

demo1.mp4

The five-node workflow

The bundled text_to_image graph illustrates five stages:

  1. Load Models selects the pipeline type and exposes compatible text encoders, denoise model, VAE, scheduler, and optional image encoder components.
  2. Encode Prompt converts prompt fields into the embeddings expected by the selected pipeline.
  3. Denoise performs the iterative generation step using width, height, step, guidance, seed, and model-specific fields.
  4. Decode Latents converts latent output into image output with the VAE.
  5. Preview Image publishes the generated image to the client/cache surface.

The exact fields and defaults come from the live registry. For example, Flux, Qwen Image, Z-Image, and Wan pipelines do not share one universal guidance, prompt, or step contract. Refresh or recreate a graph when a model's dynamic definition changes.

demo2_change_model_type.mp4

Type a prompt, confirm model readiness, and use Run. A queued task response only confirms submission; watch Queue and WebSocket progress for completion or structured failure details.

demo2_t2i.mp4

Reusing a loaded model

Compatible tasks can share components from one Load Models node. For example, an image-edit path can add image encoding/conditioning nodes while reusing the model components already loaded for text-to-image.

Component reuse depends on compatible pipeline contracts and current cache state. It reduces duplicate loading but does not guarantee that every model remains resident or that a new task avoids additional allocations.

demo3_extend_the_wokflow.mp4

Dynamic Block

Dynamic Block combines a compatible Modular Diffusers block configuration into one graph node. Enter a supported repository ID, load its definition, inspect the generated fields, and connect any required shared components or media inputs.

The shipped example uses diffusers/FLUX.2-klein-4B-modular at the immutable revision recorded in data/model-artifact-catalog.json. Repository availability and code can change; custom repositories still require an explicitly reviewed 40-character commit revision.

Dynamic blocks are not arbitrary no-code plugins. They must expose a structure understood by the current Diffusers/MoDiff integration, may require remote Python code, and can fail when upstream APIs or model files change.

Combining workflows

Multiple Modular Diffusers paths can coexist on one canvas and share compatible components. A second block can consume the output of the first while reusing loaders, or separate denoise/decode paths can compare schedulers and resource modes.

Only nodes connected to the submitted graph path execute, but shared component state still consumes memory. Inspect Queue, loader diagnostics, and GPU-process information when a combined graph exceeds available resources.

Custom Hub blocks

MoDiff can load compatible custom blocks from the Hugging Face Hub. This is a trust-sensitive feature:

Custom block repositories must publish MoDiff's current modiff_pipeline_config.json schema. The loader does not fall back to earlier extension schemas or filenames.

  1. Review the repository, owner, dependencies, license, and exact commit.
  2. Enter the reviewed 40-character commit revision; moving branches and tags are rejected.
  3. Enable trust_remote_code only when the repository requires it and you accept that its Python executes with backend-process permissions.
  4. Test on a dedicated local environment without sensitive files in work_dir.
demo6_gemini.mp4

Additional nodes

Load Models and Load Model

Load Models constructs the component set for a known pipeline configuration. Load Model can replace or provide an individual component. Hub and local sources are supported where the selected loader contract allows them.

Local folders discovered by diagnostics are not automatically runnable model packages. The loader still needs the expected Diffusers configuration, files, revision, and component type.

Quantization Config

Quantization nodes create explicit configuration objects consumed by compatible loaders. Available modes depend on the installed quantization extra, platform markers, PyTorch/CUDA versions, model architecture, and loader implementation.

Quantization can reduce memory but may introduce unsupported kernels, longer loading, quality changes, or serialization constraints. Keep a non-quantized/smaller-model fallback rather than assuming every advertised mode works on every GPU.

LoRA

LoRA nodes load a compatible adapter and connect it to loader inputs that declare LoRA support. Base model, adapter architecture, target components, and scale must be compatible. A Hub search result alone does not establish that compatibility.

Pipeline documentation

Modular nodes expose a Doc output where the upstream block provides documentation. Connect it to Data Viewer to inspect active blocks and accepted inputs. Treat live /nodes metadata as the executable contract when prose and the installed upstream revision disagree.

Image comparison

Use the image comparison node to inspect outputs from different prompts, guiders, schedulers, resource modes, or model revisions. Record the model revision, seed, graph, and settings when the comparison is intended as reproducible evidence.

compare_image.mp4

Troubleshooting

  • Missing embeddings or dynamic fields usually mean the graph definition is stale or an upstream node did not produce its connected output. Update/recreate the graph after registry refresh.
  • Missing scheduler/config errors require the compatible scheduler connection expected by the selected pipeline.
  • CUDA OOM and unsupported quantized-kernel errors need a safer model/resource plan, not repeated blind retries.
  • An unavailable optional package should be installed through the matching extra, followed by a backend restart.

See docs/troubleshooting.md for backend, accelerator, model-download, and bundle diagnostics.