Add Differential Diffusion to HunyuanDiT.

[MnCSSJ4x]

What does this PR do?

Adds Differential Diffusion to HunyuanDIT.

Fixes Partially #8924(HunyuanDiT Only)

Before submitting

• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?

How to test:

Gradient

importtorchfromdiffusersimportFlowMatchEulerDiscreteSchedulerfromdiffusers.utilsimportload_imagefromPILimportImagefromtorchvisionimporttransformsfrompipeline_hunyuandit_differential_img2imgimport (
HunyuanDiTDifferentialImg2ImgPipeline,
)
pipe=HunyuanDiTDifferentialImg2ImgPipeline.from_pretrained(
"Tencent-Hunyuan/HunyuanDiT-Diffusers", torch_dtype=torch.float16
).to("cuda")
source_image=load_image(
"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png"
)
map=load_image(
"https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask_2.png"
)
prompt="a green pear"negative_prompt="blurry"image=pipe(
prompt=prompt,
negative_prompt=negative_prompt,
image=source_image,
num_inference_steps=28,
guidance_scale=4.5,
strength=1.0,
map=map,
).images[0]

Gradient	Input	Output

A colab notebook demonstrating all results can be found here. Depth Maps have also been added in the same colab.

Who can review?

@a-r-r-o-w@DN6

[a-r-r-o-w]

Thank you for working on this, looks good to me! I think we can merge this once you add your name and contrib to the community README file. Also, seems like at place of the code, the style guide is not followed. Normally, these could be fixed with make style if it was a pipeline in core diffusers. However, since this is a community pipeline, you can run styling with:
ruff check examples/community/pipeline_hunyuandit_differential_img2img.py --fix

[a-r-r-o-w]

Thank you for working on this, looks good to me! I think we can merge this once you add your name and contrib to the community README file. Also, seems like at place of the code, the style guide is not followed. Normally, these could be fixed with make style if it was a pipeline in core diffusers. However, since this is a community pipeline, you can run styling with:
ruff check examples/community/pipeline_hunyuandit_differential_img2img.py --fix

[MnCSSJ4x]

@a-r-r-o-w I have fixed the style issues and added the details to the markdown file. If all looks good, you can go ahead and merge this request.

[a-r-r-o-w]

@MnCSSJ4x Could you revert all the other changes apart from adding your name and contribution to community README? If you'd like to refactor, you can do it in a separate PR as it's out of scope for this one. Please keep the changes here limited

[MnCSSJ4x]

@MnCSSJ4x Could you revert all the other changes apart from adding your name and contribution to community README? If you'd like to refactor, you can do it in a separate PR as it's out of scope for this one. Please keep the changes here limited

Sure. I'll try to revert in fix. I feel some tool might have auto refactored it.

[MnCSSJ4x]

@a-r-r-o-w Can you please check and let me know if it's ok now? Apologies for bothering you with such trivial issues.

[a-r-r-o-w]

No problem, but there are still a large number of differences. Please run git restore -s main examples/community/README.md first, and then add your name to the table, and an example of how to run Hunyuan DiffDiff. You will know the PR is ready if it contains only additions here. Currently, it contains many deletions due to additional changes

[MnCSSJ4x]

@a-r-r-o-w Thanks for the command. It should be resolved now.

[a-r-r-o-w]

@MnCSSJ4x Looking good implementation-wise. The quality tests seems to be failing. Could you run make style and ensure that the tests pass locally too.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[MnCSSJ4x]

@MnCSSJ4x Looking good implementation-wise. The quality tests seems to be failing. Could you run make style and ensure that the tests pass locally too.

Upon running make style I get the following and am unable to track back where exactly things are going wrong or if I introduced this issues.

ruffcheckexamplesscriptssrctestsutilsbenchmarkssetup.py--fixsrc/diffusers/configuration_utils.py:679:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|677|iffield.nameinself._flax_internal_args:
678|continue679|if type(field.default) ==dataclasses._MISSING_TYPE:
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721680|default_kwargs[field.name] =None681|else:
|tests/models/test_modeling_common.py:338:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|337|model.set_default_attn_processor()
338|assertall(type(proc) ==AttnProcessorNPUforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721339|withtorch.no_grad():
340|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:346:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|345|model.enable_npu_flash_attention()
346|assertall(type(proc) ==AttnProcessorNPUforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721347|withtorch.no_grad():
348|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:354:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|353|model.set_attn_processor(AttnProcessorNPU())
354|assertall(type(proc) ==AttnProcessorNPUforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721355|withtorch.no_grad():
356|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:389:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|388|model.set_default_attn_processor()
389|assertall(type(proc) ==AttnProcessorforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^E721390|withtorch.no_grad():
391|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:397:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|396|model.enable_xformers_memory_efficient_attention()
397|assertall(type(proc) ==XFormersAttnProcessorforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721398|withtorch.no_grad():
399|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:405:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|404|model.set_attn_processor(XFormersAttnProcessor())
405|assertall(type(proc) ==XFormersAttnProcessorforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721406|withtorch.no_grad():
407|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:433:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|431|return432|433|assertall(type(proc) ==AttnProcessor2_0forprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721434|withtorch.no_grad():
435|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:441:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|440|model.set_default_attn_processor()
441|assertall(type(proc) ==AttnProcessorforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^E721442|withtorch.no_grad():
443|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:449:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|448|model.set_attn_processor(AttnProcessor2_0())
449|assertall(type(proc) ==AttnProcessor2_0forprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721450|withtorch.no_grad():
451|ifself.forward_requires_fresh_args:
|tests/models/test_modeling_common.py:457:20: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|456|model.set_attn_processor(AttnProcessor())
457|assertall(type(proc) ==AttnProcessorforprocinmodel.attn_processors.values())
|^^^^^^^^^^^^^^^^^^^^^^^^^^^E721458|withtorch.no_grad():
459|ifself.forward_requires_fresh_args:
|tests/pipelines/controlnet/test_controlnet_sdxl.py:1022:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|1021|controlnet=ControlNetModel.from_unet(unet, conditioning_channels=4)
1022|asserttype(controlnet.mid_block) ==UNetMidBlock2D|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E7211023|assertcontrolnet.conditioning_channels==4|tests/pipelines/test_pipelines_common.py:777:21: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|775|ifhasattr(component, "attn_processors"):
776|assertall(
777|type(proc) ==AttnProcessorforprocincomponent.attn_processors.values()
|^^^^^^^^^^^^^^^^^^^^^^^^^^^E721778| ), "`from_pipe` changed the attention processor in original pipeline."|tests/schedulers/test_schedulers.py:827:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|825|scheduler_loaded=DDIMScheduler.from_pretrained(f"{USER}/{self.repo_id}")
826|827|asserttype(scheduler) ==type(scheduler_loaded)
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721828|829|# Reset repo|tests/schedulers/test_schedulers.py:838:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|836|scheduler_loaded=DDIMScheduler.from_pretrained(f"{USER}/{self.repo_id}")
837|838|asserttype(scheduler) ==type(scheduler_loaded)
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721839|840|# Reset repo|tests/schedulers/test_schedulers.py:854:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|852|scheduler_loaded=DDIMScheduler.from_pretrained(self.org_repo_id)
853|854|asserttype(scheduler) ==type(scheduler_loaded)
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721855|856|# Reset repo|tests/schedulers/test_schedulers.py:865:16: E721Use`is`and`is not`fortypecomparisons, or`isinstance()`forisinstancechecks|863|scheduler_loaded=DDIMScheduler.from_pretrained(self.org_repo_id)
864|865|asserttype(scheduler) ==type(scheduler_loaded)
|^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^E721866|867|# Reset repo|Found17errors.
make: *** [style] Error1

[a-r-r-o-w]

Is your ruff version same as the one in our setup.py? I remember seeing something in the past due to incompatible ruff versions

[MnCSSJ4x]

Is your ruff version same as the one in our setup.py? I remember seeing something in the past due to incompatible ruff versions

Hi, yes the version was different. Fixed it and ran the command make style, I think the formatting was fixed and I pushed the same.

However, Got some error text

ruff check examples scripts src tests utils benchmarks setup.py --fix
ruff format examples scripts src tests utils benchmarks setup.py
1 file reformatted, 1036 files left unchanged
doc-builder style src/diffusers docs/source --max_len 119
make: doc-builder: No such file or directory
make: *** [style] Error 1

[a-r-r-o-w]

Thank you for your contribution and bearing with our reviews! This is a very strong good-first-issue finish 🎉