[bug]: Workflow - "Latents to Image" - Errors if only CPU used.

[schoenid]

Is there an existing issue for this?

• I have searched the existing issues

OS

Linux
(Ubuntu Studio 22.04, KDE Plasma Version: 5.24.7, Memory: 62.5 GiB of RAM)

GPU

cuda
(NVIDIA GeForce RTX 3060/PCIe/SSE2)

VRAM

8 GiB

What version did you experience this issue on?

3.1.1

What happened?

Created a workflow like suggested by InvokeAI - Workflow Fundamentals - Creating with Generative AI.
Everything works fine using the GPU.

After swiching the settings to CPU

I've got errors:

The image generation stops and the "Latents to Image" module reports a "nodeExecutionState.error".

So I'm required to use the CPU for calculation, if I want to do something with SDXL, because the system is also on error, if the GPU is used for SDXL. (see #4573)

Just for completion:
Trying to do something with the GPU leads to the following error:

So I need to change the settings to CPU.

Screenshots

No response

Additional context

No response

Contact Details

No response

[schoenid]

Well, found that I have to activate Fp32.

By default it is deactivated.

After Activation the whole workflow works:

The report could be closed, either if the module works on Fp16 or the Fp32 switch is activated by default.

[Millu]

This is issue is due to the default SDXL VAE that will cause an error with fp16/half precision. If you use the VAE node and install/load the sdxl-vae-fp16-fix it should work as expected.

Glad you got it sorted in any case!

[psychedelicious]

For better or worse, this is a setting you just need to be aware of and manage. Depending on the VAE in use, you may need FP16 or FP32.

On the node editor, the final LatentsToImage node is not aware of what VAE you have chosen until LatentsToImage is executed. And because this is a setting that users may wish to change arbitrarily, we won't set it automatically.

I'll close this issue as there's nothing to fix, but acknowledge that we may need more documentation around precision.

[schoenid]

This is issue is due to the default SDXL VAE that will cause an error with fp16/half precision. If you use the VAE node and install/load the sdxl-vae-fp16-fix it should work as expected.

Thank you, @Millu

As I understand the concept of invoke ai, the vae model should be installed in :
<invokeai-location>/autoimport/vae/
or by using the Import Models page of the UI.

I don't understand, how to do that with sdxl-vae-fp16-fix, especially the part with

...
Just load this checkpoint via AutoencoderKL:
...

So it's just useless for me.

[schoenid]

@psychedelicious Sorry, but I think there is something to fix:

The Fp32 switch should be activated by default on the "Latents to Image" module, because it's not working without in a standard installation. Otherwise the most Newbies will be trapped by this.

If then someone disables it, it's his own mistake.

[psychedelicious]

@schoenid fp32 is only needed for the "official" sdxl VAE (with baked-in watermarking). Is this what you mean by standard installation? It's also arguable that the standard installation uses the fp16 capable sdxl vae.

@keturn we have the default VAE precision selected on application startup using the choose_precision() helper method in invocations/latent.py. Is this correct? Should it instead default to fp32?

I think fp16 vaes will cast themselves to fp32 if needed, is that correct?

Until recently, there was no need to change vae precision manually, and we could safely use fp16 when the system supported it.

Unfortunately because stability has integrated a fp32-only vae in the sdxl model, the default precision logic is not correct when that particular vae is in use.

[keturn]

IMHO the way it should work (not necessarily the current implementation) is that the precision is chosen by:

1. Default to the global InvokeAI.Device.Precision configuration setting.
2. If you have a model that cannot run in that precision, that should be specified in the model's config (either its own config.json or in its entry in our configs/models.yaml configuration), and the model loader is responsible for using that setting to override the default.
3. There is no 3. There's nearly never a reason to change this per-generation or per-workflow; it should be solely a property of the model, not the encode/decode node inputs.

and, practically speaking, nobody should distribute models that can't run in fp16, because we ain't all got A100s out here with memory to spare.

I think fp16 vaes will cast themselves to fp32 if needed, is that correct?

Ehhmmm..... there does exist a force_upcast configuration option on the VAE model class: https://huggingface.co/docs/diffusers/api/models/autoencoderkl#diffusers.AutoencoderKL.force_upcast

but as far as I can tell, that's not actually implemented by said class, it relies on the supporting pipeline to honor it. Which Invoke's model manager and LatentsToImageInvocation function probably isn't doing, because I'm not finding that string in any of our python sources.

and even when the fp32 toggle is on LatentsToImageInvocation it messes around with the internals of the VAE to make some parts not actually fp32 after all:

InvokeAI/invokeai/app/invocations/latent.py

Lines 668 to 671 in 1419977

	if use_torch_2_0_or_xformers:
	vae.post_quant_conv.to(latents.dtype)
	vae.decoder.conv_in.to(latents.dtype)
	vae.decoder.mid_block.to(latents.dtype)

(also if you need more dynamic range than fp16 and your hardware supports it, consider bfloat16 instead of float32.)

does that clear things up?

[Millu]

@schoenid to install the fp16 vae, paste this into the "Model Location" field of the Import Models tab: madebyollin/sdxl-vae-fp16-fix and press "Add Model"

[psychedelicious]

does that clear things up?

lol, clear as one step of LMS with "mud" as the positive prompt.

So, followup - acknowledging that there are deeper changes to make, and ignoring them for now - should the LatentsToImage node default fp32?

[psychedelicious]

(re-opening this as it's now a discussion)

[schoenid]

fp32 is only needed for the "official" sdxl VAE (with baked-in watermarking). Is this what you mean by standard installation? It's also arguable that the standard installation uses the fp16 capable sdxl vae.

So as a Newbie I've installed invoke AI a few days ago like described at https://github.com/invoke-ai/InvokeAI
It was completely new for me.
Then I did the tutorial for workflows from https://www.youtube.com/watch?v=L3isi26qy0Y
I also found the libraries at civitai.com and tried to replicate some of the pictures, mostly done with SDXL.
Suddenly I run into problems with CUDA (not enogh memory) and decided only to use the CPU for generation, altough it's much slower. However I wanted to use a workflow, which produced the known Errors from this post.

I didn't install any model until this time.
That's what I mean with "standard installation".
The only thing I've changed was the setting to use the CPU (which of course is not standard ...).

So, followup - acknowledging that there are deeper changes to make, and ignoring them for now - should the LatentsToImage node default fp32?

Yes.
It would have spared me much time and effort to solve the problem.
On the other hand, I've learned a lot in the short time, thanks to the problem. ;)
Other people would maybe not, because they mostly give up.

By the way:
I solved the CUDA problem, simply by manually reducing the amount of VRAM cache to ~700MB (from 1.4 of 8GiB).
For some models it still seems to be to much ...

[psychedelicious]

You can now set VAE precision as a model default setting on a per-model basis.

The node editor will continue to have VAE Precision as a manual setting.