error: input types 'tensor<2x1280xf32>' and 'tensor<*xf16>' are not broadcast compatible

[toffaletti]

dream.py crashes with this error on my M1. The original scripts work fine.

I've stepped through with a debugger and traced the problem to this, but I'm a bit out of my area here.
https://github.com/lstein/stable-diffusion/blob/2aa83932728242b05248fada8447506e2fbed9f0/ldm/modules/diffusionmodules/openaimodel.py#L798

ldm/modules/diffusionmodules/openaimodel.py

    def forward(self, x, timesteps=None, context=None, y=None, **kwargs):
        """
        Apply the model to an input batch.
        :param x: an [N x C x ...] Tensor of inputs.
        :param timesteps: a 1-D batch of timesteps.
        :param context: conditioning plugged in via crossattn
        :param y: an [N] Tensor of labels, if class-conditional.
        :return: an [N x C x ...] Tensor of outputs.
        """
        assert (y is not None) == (
            self.num_classes is not None
        ), 'must specify y if and only if the model is class-conditional'
        hs = []
        t_emb = timestep_embedding(
            timesteps, self.model_channels, repeat_only=False
        )
        emb = self.time_embed(t_emb) <----- crashes when stepping into this line

        self.time_embed = nn.Sequential(
            linear(model_channels, time_embed_dim),
            nn.SiLU(),
            linear(time_embed_dim, time_embed_dim),
        )

I am working from this set of changes just to get this far: https://github.com/toffaletti/stable-diffusion/tree/dream-m1

[JimDabell]

I’m not sure why half precision isn’t working, but you should be able to work around it by using the --full_precision command-line argument.

[toffaletti]

Awesome, that worked and I'm now able to use dream.py on M1 with the changes in the branch I linked above, which I'll open a PR for soon. How should I address this requirement to use --full_precision? Do we just want to document it? Spit out a log message? Force it on if using MPS device? The problem besides it just not functioning without --full_precision is that it just crashes the process (abort style) which then prints a bunch of other warnings about leaked multiprocess semaphores.

[Vargol]

I'm a firm beaver that default setting should work so I think full_precision should be forced on M1 or for MPS if CUDA is not available , as I guess that as some Macs with NVIDIA cards for which CUDA works was well as MPS.

[lstein]

It is perfectly reasonable to default to full precision on M1 hardware. Perhaps the devs working on PR #268 can fold this in.

[toffaletti]

It is perfectly reasonable to default to full precision on M1 hardware. Perhaps the devs working on PR #268 can fold this in.

Already addressed in that PR