[Community] Add pipeline for CLIPSeg x Stable Diffusion

[NielsRogge]

Model/Pipeline/Scheduler description

We've just added CLIPSeg to the 🤗 Transformers library, making it possible to use CLIPSeg in a few lines of code as shown in this notebook. The model is a minimal extension of CLIP for zero-shot and one-shot image segmentation.

It'd be great to create a new pipeline that leverages it for text-based (prompt) image inpainting. This way, people can just type whatever they want to inpaint in an image with a model like Stable Diffusion.

The idea of leveraging CLIPSeg was proposed here: https://github.com/amrrs/stable-diffusion-prompt-inpainting.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

CLIPSeg is available here: https://huggingface.co/docs/transformers/main/en/model_doc/clipseg.

[dblunk88]

nice! This could be combined with inpainting to potentially (more accurately) replace existing things in an image.

[WASasquatch]

I still love the idea of this, but the detection and mask creation is still an inherent problem. Sure it looks nice in your example against a white background, but anything else, and the masking is clearly visible in the diffusions in painting in anomalies with it's boundary. This needs some sort of algorithm to create curved points between edges.

[nokunato]

Hi novice here, but can't seem to use diffusers when making use of this CLIPseg. Is there anyway around this. The command down below doesn't seem to run when using "from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation". Help

device = "cuda"
pipe = StableDiffusionInpaintPipeline.from_pretrained(
"CompVis/stable-diffusion-v1-4",
revision="fp16",
torch_dtype=torch.float16,
use_auth_token=True
).to(device)

[unography]

Hi @patrickvonplaten , I added a PR for this here - #1250

[WASasquatch]

I wonder if CLIPSeg can be improved with Guassian Blur and SVG Tracing, producing a two tone mask that has curved tracing based on Gaussian smoothing. Has this been suggested to CLIPSeg? I really want something like this in diffusers, but also want it to be complementary with art and fidelity.

[tenghui98]

Hi novice here, but can't seem to use diffusers when making use of this CLIPseg. Is there anyway around this. The command down below doesn't seem to run when using "from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation". Help

device = "cuda" pipe = StableDiffusionInpaintPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ).to(device)

Hi novice here, but can't seem to use diffusers when making use of this CLIPseg. Is there anyway around this. The command down below doesn't seem to run when using "from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation". Help

device = "cuda" pipe = StableDiffusionInpaintPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ).to(device)

How fix this?

[nokunato]

Hi novice here, but can't seem to use diffusers when making use of this CLIPseg. Is there anyway around this. The command down below doesn't seem to run when using "from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation". Help
device = "cuda" pipe = StableDiffusionInpaintPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ).to(device)

Hi novice here, but can't seem to use diffusers when making use of this CLIPseg. Is there anyway around this. The command down below doesn't seem to run when using "from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation". Help
device = "cuda" pipe = StableDiffusionInpaintPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ).to(device)

How fix this?

Not sure, tried but getting this new error after trying to download the inpaint model

new error: "ValueError: The component <class 'transformers.models.clip.image_processing_clip.CLIPImageProcessor'> of <class 'diffusers_modules.git.text_inpainting.TextInpainting'> cannot be loaded as it does not seem to have any of the loading methods defined in {'ModelMixin': ['save_pretrained', 'from_pretrained'], 'SchedulerMixin': ['save_config', 'from_config'], 'DiffusionPipeline': ['save_pretrained', 'from_pretrained'], 'OnnxRuntimeModel': ['save_pretrained', 'from_pretrained'], 'PreTrainedTokenizer': ['save_pretrained', 'from_pretrained'], 'PreTrainedTokenizerFast': ['save_pretrained', 'from_pretrained'], 'PreTrainedModel': ['save_pretrained', 'from_pretrained'], 'FeatureExtractionMixin': ['save_pretrained', 'from_pretrained']}."

[misbahsy]

Getting the same error as commented by @nokunato . Any fix @NielsRogge ?

[NielsRogge]

Have you installed Transformers from source?

[misbahsy]

Have you installed Transformers from source?

yes, I did. Tried pip install, as well as, git install -
git+https://github.com/huggingface/transformers.git

Let me know if I missed something. Thanks.