[bug]: CUDA out of memory error with 3.1.1. No previous issue with 3.1.0.

[JakeSSRN]

Is there an existing issue for this?

• I have searched the existing issues

OS

Windows

GPU

cuda

VRAM

8 GB

What version did you experience this issue on?

3.1.1

What happened?

I'm getting the typical 'CUDA out of memory' error when generating with SDXL and SD2.1. With SD2.1 I even generate at 512x512 and 1 image batch size. In 3.1.0 I could generate a batch size of 3 at 1024x1024 with SDXL without running into problems. What gives?

System RAM cache is set to 12 of 16 GB and VRAM cache is set to 0 of 8 GB.

I noticed that the "free GPU memory after every generation" option is gone. I suspect this has something to do with it. Also, the whole browser crashes as soon as the error is thrown and I have to restart both the browser and Invoke to get it working again.

I've posted in the help section of the discord two days in a row now with absolutely zero response. I'd appreciate some insight, please.

Screenshots

Additional context

Probably unrelated: how I do I get Invoke to stop opening an new tab with the asset images in it every time I generate with the unified canvas?

Contact Details

No response

[Millu]

Sorry you message was missed on Discord - just responded there!

[Petterson19]

I'm with the same problem and I can't join to discord

[schoenid]

Same to me ... except I'm on Ubuntu 22.04.
Seems not to be a model problem. If I use the CPU only, it works, but it needs just 10+ times longer.

[Petterson19]

Same to me ... except I'm on Ubuntu 22.04. Seems not to be a model problem. If I use the CPU only, it works, but it needs just 10+ times longer.

I found a solution, just downloaded CUDA from Nvidia and it's running smoothly now!

[schoenid]

I found a solution, just downloaded CUDA from Nvidia and it's running smoothly now!

@Petterson19 Thank you for the hint!

Unfortunately I've got it already installed.

~~>>> wrong <<< On SDXL: I can use CUDA up to a resolution of ~256x256. ~~

• On SD1: up to 1280x720.
• On SD2: up to 512x512

For higher resolutions: I have to switch to CPU only.

"nvidi-smi" from the terminal prints me the status of the GPU to the screen, with a list of all processes, which are using the GPU. So I can close any process, which is not really needed. It helps to increase the maximum resolution a little bit.

[JakeSSRN]

I found a solution, just downloaded CUDA from Nvidia and it's running smoothly now!

I uninstalled and reinstalled both CUDA toolkit and visual studio and restarted in between each step. No improvement.

[Petterson19]

I found a solution, just downloaded CUDA from Nvidia and it's running smoothly now!

I uninstalled and reinstalled both CUDA toolkit and Visual Studio and restarted in between each step. No improvement.

Why didn't you use phyton in prompt or PowerShell? just too easy to install it

[schoenid]

So I made a mistake. The selected model I thougt was SDXL was SD2.1 only.

On SDXL and CUDA enabled I've got always Memory Errors:

[2023-09-21 00:36:29,950]::[InvokeAI]::ERROR --> Error while invoking:
CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.78 GiB total capacity; 5.80 GiB already allocated; 61.69 MiB free; 6.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hm, the message means: 61.69 MiB are free, but 26.00 MiB can't be allocated. (Why ???)
The PYTORCH_CUDA_ALLOC_CONF is a matter for the developer, as I see. I can't do this setting.

So SDXL is not usable on CUDA in invoke ai v3.1.1.

Remark
nvidia-smi (CUDA-information):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   50C    P3    37W / 170W |   7879MiB /  8192MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1443      G   /usr/lib/xorg/Xorg                284MiB |
|    0   N/A  N/A      2268      G   /usr/bin/kwin_x11                  66MiB |
|    0   N/A  N/A      2316      G   /usr/bin/plasmashell               94MiB |
|    0   N/A  N/A      2785      G   ...402625461174669194,262144      166MiB |
|    0   N/A  N/A     55892      C   ...keAI/.venv/bin/python3.10     7260MiB |
+-----------------------------------------------------------------------------+

Processe:

• First 3 is the system
• 4th is my browser (Opera / required for UI)
• 5th is invoke ai

[JakeSSRN]

The PYTORCH_CUDA_ALLOC_CONF is a matter for the developer, as I see. I can't do this setting.
So SDXL is not usable on CUDA in invoke ai v3.1.1

I'm at a loss here. I guess I'll just have to wait for an update.

[JakeSSRN]

In a last ditch effort, I did a clean install of Windows 11. Invoke v3.1.1 and SDXL now work without issue. Still no idea what the problem was.

[schoenid]

I didn't realize, that the amount of used GPU memory (VRAM) can be reduced in setup:

So easy!
Now everything works fine!

[icamps]

Having this same error with InvokeAI 4.2.7.

No option in setup to reduce the amount of memory used, either option to "free GPU memory after every generation".

My box is running Ubuntu 22.04 and CUDA 12.4.

[Harvester62]

Because we do not maintain older versions of Invoke, is this still a relevant open issue? Has the user tested with the latest currently available release (v5.4.2)?