feat: add Mixture-of-Diffusers ControlNet Tile upscaler Pipeline for SDXL

[elismasilva]

What does this PR do?

This PR implements a community pipeline that leverages ControlNet Tile and Mixture-of-Diffusers techniques, integrating tile diffusion directly into the latent space denoising process. Designed to overcome the limitations of conventional pixel-space tile processing, this pipeline delivers Super Resolution (SR) upscaling for higher-quality images, reduced processing time, and greater adaptability.

See Gradio Demo:


More details of implementation
https://github.com/DEVAIEXP/mod-control-tile-upscaler-sdxl

Local reproduction

importtorchfromdiffusersimportControlNetUnionModel, AutoencoderKL, UniPCMultistepSchedulerfrommod_controlnet_tile_sr_sdxlimportStableDiffusionXLControlNetTileSRPipelinefromdiffusers.utilsimportload_imagefromPILimportImagedevice="cuda"# Initialize the models and pipelinecontrolnet=ControlNetUnionModel.from_pretrained(
"brad-twinkl/controlnet-union-sdxl-1.0-promax", torch_dtype=torch.float16
).to(device=device)
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16).to(device=device)
model_id="SG161222/RealVisXL_V5.0"pipe=StableDiffusionXLControlNetTileSRPipeline.from_pretrained(
model_id, controlnet=controlnet, vae=vae, torch_dtype=torch.float16, use_safetensors=True, variant="fp16"
).to(device)
#pipe.enable_model_cpu_offload() # << Enable this if you have limited VRAMpipe.enable_vae_tiling() # << Enable this if you have limited VRAMpipe.enable_vae_slicing() # << Enable this if you have limited VRAM# Set selected schedulerpipe.scheduler=UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# Load imagecontrol_image=load_image("https://huggingface.co/datasets/DEVAIEXP/assets/resolve/main/1.jpg")
original_height=control_image.heightoriginal_width=control_image.widthprint(f"Current resolution: H:{original_height} x W:{original_width}")
# Pre-upscale image for tilingresolution=4096tile_gaussian_sigma=0.3max_tile_size=1024# or 1280current_size=max(control_image.size)
scale_factor=max(2, resolution/current_size)
new_size= (int(control_image.width*scale_factor), int(control_image.height*scale_factor))
image=control_image.resize(new_size, Image.LANCZOS)
# Update target height and widthtarget_height=image.heighttarget_width=image.widthprint(f"Target resolution: H:{target_height} x W:{target_width}")
# Calculate overlap sizenormal_tile_overlap, border_tile_overlap=pipe.calculate_overlap(target_width, target_height)
# Set other paramstile_weighting_method=pipe.TileWeightingMethod.COSINE.valueguidance_scale=4num_inference_steps=35denoising_strenght=0.65controlnet_strength=1.0prompt="high-quality, noise-free edges, high quality, 4k, hd, 8k"negative_prompt="blurry, pixelated, noisy, low resolution, artifacts, poor details"# Image generationgenerated_image=pipe(
image=image,
control_image=control_image,
control_mode=[6],
controlnet_conditioning_scale=float(controlnet_strength),
prompt=prompt,
negative_prompt=negative_prompt,
normal_tile_overlap=normal_tile_overlap,
border_tile_overlap=border_tile_overlap,
height=target_height,
width=target_width,
original_size=(original_width, original_height),
target_size=(target_width, target_height),
guidance_scale=guidance_scale, strength=float(denoising_strenght),
tile_weighting_method=tile_weighting_method,
max_tile_size=max_tile_size,
tile_gaussian_sigma=float(tile_gaussian_sigma),
num_inference_steps=num_inference_steps,
)["images"][0]
generated_image .save("result.png")

Running after published

importtorchfromdiffusersimportDiffusionPipeline, ControlNetUnionModel, AutoencoderKL, UniPCMultistepScheduler, UNet2DConditionModelfromdiffusers.utilsimportload_imagefromPILimportImagedevice="cuda"# Initialize the models and pipelinecontrolnet=ControlNetUnionModel.from_pretrained(
"brad-twinkl/controlnet-union-sdxl-1.0-promax", torch_dtype=torch.float16
).to(device=device)
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16).to(device=device)
model_id="SG161222/RealVisXL_V5.0"pipe=DiffusionPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
vae=vae,
controlnet=controlnet,
custom_pipeline="mod_controlnet_tile_sr_sdxl", use_safetensors=True,
variant="fp16",
).to(device)
unet=UNet2DConditionModel.from_pretrained(model_id, subfolder="unet", variant="fp16", use_safetensors=True)
#pipe.enable_model_cpu_offload() # << Enable this if you have limited VRAMpipe.enable_vae_tiling() # << Enable this if you have limited VRAMpipe.enable_vae_slicing() # << Enable this if you have limited VRAM# Set selected schedulerpipe.scheduler=UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# Load imagecontrol_image=load_image("https://huggingface.co/datasets/DEVAIEXP/assets/resolve/main/1.jpg")
original_height=control_image.heightoriginal_width=control_image.widthprint(f"Current resolution: H:{original_height} x W:{original_width}")
# Pre-upscale image for tilingresolution=4096tile_gaussian_sigma=0.3max_tile_size=1024# or 1280current_size=max(control_image.size)
scale_factor=max(2, resolution/current_size)
new_size= (int(control_image.width*scale_factor), int(control_image.height*scale_factor))
image=control_image.resize(new_size, Image.LANCZOS)
# Update target height and widthtarget_height=image.heighttarget_width=image.widthprint(f"Target resolution: H:{target_height} x W:{target_width}")
# Calculate overlap sizenormal_tile_overlap, border_tile_overlap=pipe.calculate_overlap(target_width, target_height)
# Set other paramstile_weighting_method=pipe.TileWeightingMethod.COSINE.valueguidance_scale=4num_inference_steps=35denoising_strenght=0.65controlnet_strength=1.0prompt="high-quality, noise-free edges, high quality, 4k, hd, 8k"negative_prompt="blurry, pixelated, noisy, low resolution, artifacts, poor details"# Image generationgenerated_image=pipe(
image=image,
control_image=control_image,
control_mode=[6],
controlnet_conditioning_scale=float(controlnet_strength),
prompt=prompt,
negative_prompt=negative_prompt,
normal_tile_overlap=normal_tile_overlap,
border_tile_overlap=border_tile_overlap,
height=target_height,
width=target_width,
original_size=(original_width, original_height),
target_size=(target_width, target_height),
guidance_scale=guidance_scale, strength=float(denoising_strenght),
tile_weighting_method=tile_weighting_method,
max_tile_size=max_tile_size,
tile_gaussian_sigma=float(tile_gaussian_sigma),
num_inference_steps=num_inference_steps,
)["images"][0]
generated_image .save("result.png")

Result

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@asomoza@sayakpaul@yiyixuxu

[elismasilva]

strange fail check, i already did make style and make quality. By the way, whenever I do it, it also adjusts files from other commits from people who forgot to do it, so I always have to undo these other corrections. I think it is necessary to create a PR that only executes make style and make quality for these pending files, as they are coming in all new PRs and causing problems when we are going to send our PR.

[asomoza]

we have a bot now that can do quality and style, I still wait to see if the original author does it first though.

And yes, I get that sometimes it tries to format other files but it's just a matter of committing/staging your relevant files only, maybe for maintainers it's easier since I'm used to have a lot of files changed but not staged.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

So I did some quick tests with a mobile 4090 and it takes for a 4k image 2m52s which is not bad, my own version takes a lot longer.

Here's a comparison with what I usually use the upscalers to do, to bring back low resolution images to something more usable, so to do a 4x upscale of a 256px image:

source	this	Aura SRv2

nevertheless, this upscaler is for using tiling which is for images over 2k or 4k.

[asomoza]

thanks!

[elismasilva]

So I did some quick tests with a mobile 4090 and it takes for a 4k image 2m52s which is not bad, my own version takes a lot longer.

Here's a comparison with what I usually use the upscalers to do, to bring back low resolution images to something more usable, so to do a 4x upscale of a 256px image:

source	this	Aura SRv2

nevertheless, this upscaler is for using tiling which is for images over 2k or 4k.

Not bad. Low resolution images take a little more work to maintain detail.

Did you use FP8 or FP16? FP8 on my 3060ti goes fast without losing quality.