Memory efficient attention not working with fp16 weights

[apolinario]

Describe the bug

Following the example code with available in the 0.7.0 release

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    revision="fp16",
    torch_dtype=torch.float16,
).to("cuda")

pipe.enable_xformers_memory_efficient_attention()

with torch.inference_mode():
    sample = pipe("a small cat")

I'm getting the following error:

RuntimeError: expected scalar type Half but found Float

System Info

diffusers==0.7.2
pytorch==1.12.1

[apolinario]

The memory efficient attention mechanism seems to be working just fine - if I do a
with torch.autocast("cuda") instead of with torch.inference_mode() it works, and performs faster with enable_xformers_memory_efficient_attention as opposed to disabled.

So it seems like a casting issue indeed as described

[nmntz]

Could it be related to this issue? #532 (comment)
Can you try updating the xformers library?