[Modular] implement requirements validation for custom blocks

[sayakpaul]

What does this PR do?

As discussed with Dhruv.

When implementing custom blocks (which we all believe will grow big :)), it can be useful for the block author and the users to have some kind of dependencies and their versions specified. This PR implements that feature.

Sample output: https://huggingface.co/diffusers-internal-dev/gemini-prompt-expander/blob/main/modular_config.json.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

@DN6 a gentle ping :)

[sayakpaul]

Gentle ping @DN6

[yiyixuxu]

ummm, should requirements be defined at the pipeline level instead of the block level?
Blocks are meant to be reusable components that can be assembled in different ways. While individual blocks can define default loading behavior (which may have different dependency rquirement), these should be overridable at the pipeline level when blocks are assembled together in modular pipeline

the person who put together the pipeline should be responsible and have the final say over the final components, e.g requirements and modular_model_index.json etc, no?

[sayakpaul]

I think implementing it at both the block and pipeline levels could be nice. What I am thinking:

• If a pipeline doesn't define any custom requirements, then we could check if any of its blocks define them and perform validation.
• If a pipeline defines it, then the validation logic, at least, still remains the same.

WDYT about this approach? 👀

[yiyixuxu]

for blocks, maybe define requirements within the block definition similar to inputs/outputs?

classGeminiPromptExpander:
_requirements= {
"google-generativeai": ">=0.8.0",
"pillow": ">=10.0.0"
}
classFluxTransformer:
_requirements= {
"transformers": ">=4.44.0",
"sentencepiece": ">=0.2.0"
}

When we combine them:

pipe=SequentialPipelineBlocks.from_blocks_dict({
"prompt_expander": GeminiPromptExpander, "transformer": FluxTransformer
})
print(pipe._requirements)
# Output:# {# "prompt_expander": {"google-generativeai": ">=0.8.0", "pillow": ">=10.0.0"},# "transformer": {"transformers": ">=4.44.0", "sentencepiece": ">=0.2.0"}# }

This should be just meta data thought, it just helps generate the final requirements.txt, but the pipeline author still need to manually write the requirement file and test it

[sayakpaul]

@yiyixuxu yes that makes sense. I will apply those changes in the PR.

[sayakpaul]

@yiyixuxu I have implemented what we discussed above. Individual ModularPipelineBlocks can implement a _requirements class-level attribute. And then SequentialPipelineBlocks now has _requirements property, which basically collates all the requirements from its blocks, if available.

fromdiffusers.modular_pipelinesimportSequentialPipelineBlocksclassGeminiPromptExpander:
_requirements= {
"google-generativeai": ">=0.8.0",
"pillow": ">=10.0.0"
}
classFluxTransformer:
_requirements= {
"transformers": ">=4.44.0",
"sentencepiece": ">=0.2.0"
}
pipe=SequentialPipelineBlocks.from_blocks_dict({
"prompt_expander": GeminiPromptExpander, "transformer": FluxTransformer
})
print(pipe._requirements)
pipe.save_pretrained(".")

LMK.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Also cc: @stevhliu. Could you help with the docs for this?

[DN6]

All good with saving requirements to the config file.

But since requirements are metadata, I don't think we need to run extensive normalization and validation. We already warn users of missing packages when trying to load custom code.

diffusers/src/diffusers/utils/dynamic_modules_utils.py

Lines 153 to 156 in 35086ac

	logger.warning(
	"This modeling file might require the following packages that were not found in your environment: "
	f"{', '.join(missing_packages)}. Run `pip install {' '.join(missing_packages)}`"
	)

[sayakpaul]

@DN6 I have updated the hub_kwargs related comments. I decided to keep the other logic as is because I don't think it would hurt to have some standardization, given that it has more benefits. For example, if a developer is including requirements then it's better if they do it in a standard way so it's better for the community to prevent reproducibility issues.

[yiyixuxu]

thanks for working on this!

[yiyixuxu]

should we add the same _requirements property to ConditionalPipelineBlocks & others?

right now, unless it's all SequentialPipelineBlocks, the nested requirements get lost, no?

block_a (conditional) ---------------------> block_b -block_a.1 -block_a.2

block_a's subblock requirement will be lost

the algo might be a bit aggresive for conditional blocks (we'd collect from all branches even though the user may only want to run one), but since this is warn-only I think that's fine

In practice, I think we'd recommend people override _requirements directly on their final block class rather than relying on the auto-accumulation, but good to have a default and have this meta data