[Pipeline] Enhancements to AutoPipeline

[suzukimain]

What does this PR do?

This PR adds the ability to automatically search for models from civitai and huggingface in AutoPipeline and enables the loading of single file checkpoints.

From #9986

Example:

pipinstallgit+https://github.com/suzukimain/diffusers.git@Update_AutoPipeline

# Search for CivitaifromdiffusersimportAutoPipelineForText2Imagepipe=AutoPipelineForText2Image.from_civitai("any").to("cuda")
image=pipe("cat").images[0]
image.save("cat.png")

# Search for HuggingfacefromdiffusersimportAutoPipelineForText2Imagepipe=AutoPipelineForText2Image.from_huggingface("any").to("cuda")
img=pipe("cat").images[0]
img.save("cat.png")

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@yiyixuxu@asomoza@bghira

[bghira]

cc @vladmandic check this out 😹 oh god @suzukimain you genius. thank you.

[vladmandic]

cc @vladmandic check this out 😹 oh god @suzukimain you genius. thank you.

i have something similar, but this is pretty clean!

needs a bit of cleanup to actually use params - e.g., instead of

model_path = f"/root/.cache/Civitai/{repo_id}/{version_id}/{file_name}"

use actual cache_dir param to construct path.
also, download resume-on-error is almost a necessity given the size of the models and how unreliable civitai can be at the times.

[yiyixuxu]

hi @suzukimain

If you make this tool as separate repo, we are happy to add it to our doc and help promote it!

[suzukimain]

hi @suzukimain

If you make this tool as separate repo, we are happy to add it to our doc and help promote it!

Originally, this PR was incorporated into AutoPipeline based on the repository below, so it is possible.
However, since it was created as a hobby project, it might be necessary to implement the changes and organize the entire repository, as well as change the license.

github: https://github.com/suzukimain/auto_diffusers

pypi: https://pypi.org/project/auto-diffusers/

[bghira]

assuming huggingface doesnt want to support a competitor even if it benefits the user or this project. bummer

[suzukimain]

assuming huggingface doesnt want to support a competitor even if it benefits the user or this project. bummer

Sorry, what do you mean by “competitor”?

[bghira]

CivitAI as a model hosting provider

[suzukimain]

Oh, I see

[yiyixuxu]

@suzukimain feel free to dress it up and write a nice introduction about it!
cc @asomoza here too! maybe we can use it to make diffusers node:)

[asomoza]

@suzukimain this is cool and I agree that it would work a lot better as an external tool with your library. Do you have plans for something similar with loras?

[suzukimain]

@suzukimain this is cool and I agree that it would work a lot better as an external tool with your library. Do you have plans for something similar with loras?

The functions search_huggingface and search_civitai, which search for hubs, are designed to support multiple tasks.
Thus, you can support different tasks by simply changing the model_type argument.
However, baseModel and types are not yet supported by the Hugging Face API at this time and could not be implemented.

Details

By the way, the baseModels should be in hf_api.model_info, but I get an error.

fromhuggingface_hubimporthf_apiinfo=hf_api.model_info(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
expand=["baseModels"]
)
print(info.baseModels)
# AttributeError: 'ModelInfo' object has no attribute 'baseModels'

diffusers/src/diffusers/pipelines/auto_pipeline.py

Lines 783 to 791 in dbe8fcb

	# Set up parameters and headers for the CivitAI API request
	params= {
	"query": search_word,
	"types": model_type,
	"sort": "Most Downloaded",
	"limit": 20,
	}
	ifbase_modelisnotNone:
	params["baseModel"] =base_model

fromdiffusers.pipelines.auto_pipelineimport (
search_huggingface,
search_civitai,
) # Search LoraLora=search_civitai(
"Keyword_to_search_Lora",
model_type="LORA",
base_model="SD 1.5",
download=True,
)
# Load Lora into the pipeline.pipeline.load_lora_weights(Lora)
# Search TextualInversionTextualInversion=search_civitai(
"EasyNegative",
model_type="TextualInversion",
base_model="SD 1.5",
download=True
)
# Load TextualInversion into the pipeline.pipeline.load_textual_inversion(TextualInversion, token="EasyNegative")

[yiyixuxu]

cc @stevhliu, where should this go if we want to add a doc page about it (as an external library)?

[suzukimain]

Made it available for use as an external tool.
The main changes are as follows:

• Changed the license to Apache-2.0, the same as Diffusers.
• Updated the downloader to huggingface_hub.http_get to support resume-on-error.
• Cache directory is now obtained by os.path.expanduser.

Made it as usable as possible in the same way as a pull request.

pipinstall--quietauto_diffusersfromauto_diffusersimportEasyPipelineForText2Image# Search for Huggingfacepipe=EasyPipelineForText2Image.from_huggingface("any").to("cuda")
img=pipe("cat").images[0]
img.save("cat.png")
# Search for Civitaipipe=EasyPipelineForText2Image.from_civitai("any").to("cuda")
image=pipe("cat").images[0]
image.save("cat.png")

@yiyixuxu, @stevhliu, @bghira

[stevhliu]

I think you can add your project here and then link to the docs you're making in #9986

[suzukimain]

I think you can add your project here and then link to the docs you're making in #9986

Based on this advice, I have made some revisions here. Is it okay to apply these changes?
I apologize for the inconvenience.

[suzukimain]

I think you can add your project here and then link to the docs you're making in #9986

Based on this advice, I have made some revisions here. Is it okay to apply these changes? I apologize for the inconvenience.

I intend to apply the changes in this way.

[stevhliu]

Yeah thats fine with me, feel free to apply the changes in #9986 :)

[suzukimain]

hi @stevhliu , What should I do with this PR?

[stevhliu]

I think this can be closed since we your project is here https://github.com/huggingface/diffusers/tree/main/examples/model_search now :)

[suzukimain]

hello @stevhliu
Understood. Also, if possible, I would appreciate it if you could let me know how this matter will turn out.

I think you can add your project here and then link to the docs you're making in #9986

[suzukimain]

Add this functionality to auto_diffusers and close this as #9986 and #10358 have been merged. thank you!