GFPGAN executing fine for 1 image at a time, but OOM when using -n parameter, is there method to clear memory in between ?

[smoke2007]

I read about GFPGAN needing more GPU memory, however I always manage to get it working with 1 generation for example for img2img, it produces a double resolution correctly, however if I used -n2 or anything higher, the next generation fails due to a Out of memory error. Is it possible to clear the memory between generations as for example, the rtx 3060 ti seems to be able to always execute it correctly for only 1 image but not in batch?

Thank you, awesome project with insanely fast developments !!

[blessedcoolant]

I read about GFPGAN needing more GPU memory, however I always manage to get it working with 1 generation for example for img2img, it produces a double resolution correctly, however if I used -n2 or anything higher, the next generation fails due to a Out of memory error. Is it possible to clear the memory between generations as for example, the rtx 3060 ti seems to be able to always execute it correctly for only 1 image but not in batch?

Thank you, awesome project with insanely fast developments !!

This error is precisely why I made the changes I did. @lstein If you were looking for a reference on the approach I took to tackle this memory issue. My PR #102 solves this issue too.

The current implementation breaks on the second iteration. There's no way to run both at the same time without causing a CUDA crash on most systems.

[lstein]

I think we're really close. Just need to slow down the pulling into main until the dust settles! Lincoln

…

On Fri, Aug 26, 2022 at 12:23 PM blessedcoolant ***@***.***> wrote: I read about GFPGAN needing more GPU memory, however I always manage to get it working with 1 generation for example for img2img, it produces a double resolution correctly, however if I used -n2 or anything higher, the next generation fails due to a Out of memory error. Is it possible to clear the memory between generations as for example, the rtx 3060 ti seems to be able to always execute it correctly for only 1 image but not in batch? Thank you, awesome project with insanely fast developments !! This error is precisely why I made the changes I did. @lstein <https://github.com/lstein> If you were looking for a reference. My PR #102 <#102> solves this issue too. — Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3EVKJNMW2WKPVZ23RRPLV3DVQLANCNFSM57XHYHZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Lincoln Stein* Head, Adaptive Oncology, OICR Senior Principal Investigator, OICR Professor, Department of Molecular Genetics, University of Toronto Tel: 416-673-8514 Cell: 416-817-8240 ***@***.*** *E**xecutive Assistant* Michelle Xin Tel: 647-260-7927 ***@***.*** ***@***.***>* *Ontario Institute for Cancer Research* MaRS Centre, 661 University Avenue, Suite 510, Toronto, Ontario, Canada M5G 0A3 @OICR_news <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Foicr_news&data=04%7C01%7CMichelle.Xin%40oicr.on.ca%7C9fa8636ff38b4a60ff5a08d926dd2113%7C9df949f8a6eb419d9caa1f8c83db674f%7C0%7C0%7C637583553462287559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PS9KzggzFoecbbt%2BZQyhkWkQo9D0hHiiujsbP7Idv4s%3D&reserved=0> | www.oicr.on.ca *Collaborate. Translate. Change lives.* This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.

[Oceanswave]

Yeah having the discussions of tradeoffs of memory and processing time! :)

Just to capture it, on the main branch, on my config, immediately after dream loads, the difference in directly GPU memory allocation of the python process is following with --gfpgan and not gfpgan.

metrics are pretty stable across multiple startups/runs, which is encouraging, so - 'cause I'm lazy - I'm not going to compute an average across runs for now.

After Startup:
Without GFPGAN: 4,976,196
With GFPGAN: 5,551,696

after I give it a prompt, memory shoots up, I use the following prompt, and specify -G1 for the with GFPGAN

a man wearing a pineapple hat -S1

T1:
Without GFPGAN: 9,207,408
With GFPGAN: 9,811,572

T2:
Without GFPGAN: 9,207,408
With GFPGAN: 9,811,572

T3:
Without GFPGAN: 9,209,456
With GFPGAN: 9,811,572

T4:
Without GFPGAN: 9,209,456
With GFPGAN: 9,811,572

T5:
Without GFPGAN: 9,209,456
With GFPGAN: 9,813,620

so pretty stable, but a little bit of allocation after the fact

So at least on my box, GFPGAN increases VRAM util by ~600MB

Keep in mind that other things on your desktop system use dedicated VRAM, so if you have a bunch of tabs open, or open up photoshop to load an image after it's done, those are going to change how much free VRAM you have available.

[Oceanswave]

More Data:

After init:

No-GFPGAN

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4084 MB |    4084 MB |    4084 MB |    8192 B  |
|       from large pool |    4039 MB |    4039 MB |    4039 MB |       0 B  |
|       from small pool |      45 MB |      45 MB |      45 MB |    8192 B  |
|---------------------------------------------------------------------------|
| Active memory         |    4084 MB |    4084 MB |    4084 MB |    8192 B  |
|       from large pool |    4039 MB |    4039 MB |    4039 MB |       0 B  |
|       from small pool |      45 MB |      45 MB |      45 MB |    8192 B  |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    4194 MB |    4194 MB |    4194 MB |       0 B  |
|       from large pool |    4148 MB |    4148 MB |    4148 MB |       0 B  |
|       from small pool |      46 MB |      46 MB |      46 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |  111981 KB |  121852 KB |    1057 MB |     948 MB |
|       from large pool |  111232 KB |  120960 KB |    1027 MB |     918 MB |
|       from small pool |     749 KB |    3142 KB |      30 MB |      29 MB |
|---------------------------------------------------------------------------|
| Allocations           |    1147    |    1148    |    1149    |       2    |
|       from large pool |     344    |     344    |     344    |       0    |
|       from small pool |     803    |     804    |     805    |       2    |
|---------------------------------------------------------------------------|
| Active allocs         |    1147    |    1148    |    1149    |       2    |
|       from large pool |     344    |     344    |     344    |       0    |
|       from small pool |     803    |     804    |     805    |       2    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     149    |     149    |     149    |       0    |
|       from large pool |     126    |     126    |     126    |       0    |
|       from small pool |      23    |      23    |      23    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      67    |      68    |     137    |      70    |
|       from large pool |      63    |      63    |     113    |      50    |
|       from small pool |       4    |      14    |      24    |      20    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

GFPGAN

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4638 MB |    4638 MB |    4702 MB |   65377 KB |
|       from large pool |    4499 MB |    4499 MB |    4499 MB |       0 KB |
|       from small pool |     138 MB |     138 MB |     202 MB |   65377 KB |
|---------------------------------------------------------------------------|
| Active memory         |    4638 MB |    4638 MB |    4702 MB |   65377 KB |
|       from large pool |    4499 MB |    4499 MB |    4499 MB |       0 KB |
|       from small pool |     138 MB |     138 MB |     202 MB |   65377 KB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    4756 MB |    4756 MB |    4756 MB |       0 B  |
|       from large pool |    4616 MB |    4616 MB |    4616 MB |       0 B  |
|       from small pool |     140 MB |     140 MB |     140 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |  120354 KB |  148211 KB |    1443 MB |    1325 MB |
|       from large pool |  119040 KB |  132864 KB |    1274 MB |    1158 MB |
|       from small pool |    1314 KB |   31465 KB |     168 MB |     167 MB |
|---------------------------------------------------------------------------|
| Allocations           |    2829    |    2829    |    3533    |     704    |
|       from large pool |     453    |     453    |     453    |       0    |
|       from small pool |    2376    |    2376    |    3080    |     704    |
|---------------------------------------------------------------------------|
| Active allocs         |    2829    |    2829    |    3533    |     704    |
|       from large pool |     453    |     453    |     453    |       0    |
|       from small pool |    2376    |    2376    |    3080    |     704    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     214    |     214    |     214    |       0    |
|       from large pool |     144    |     144    |     144    |       0    |
|       from small pool |      70    |      70    |      70    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      67    |     104    |     361    |     294    |
|       from large pool |      63    |      64    |     130    |      67    |
|       from small pool |       4    |      41    |     231    |     227    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

a man wearing a pineapple hat -S1 -n2

No-GFPGAN

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4085 MB |    6199 MB |    2079 GB |    2075 GB |
|       from large pool |    4039 MB |    6152 MB |    2063 GB |    2059 GB |
|       from small pool |      45 MB |      52 MB |      15 GB |      15 GB |
|---------------------------------------------------------------------------|
| Active memory         |    4085 MB |    6199 MB |    2079 GB |    2075 GB |
|       from large pool |    4039 MB |    6152 MB |    2063 GB |    2059 GB |
|       from small pool |      45 MB |      52 MB |      15 GB |      15 GB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    7348 MB |    7348 MB |    7348 MB |       0 B  |
|       from large pool |    7294 MB |    7294 MB |    7294 MB |       0 B  |
|       from small pool |      54 MB |      54 MB |      54 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |  113503 KB |    1518 MB |    3475 GB |    3475 GB |
|       from large pool |  111232 KB |    1516 MB |    3454 GB |    3453 GB |
|       from small pool |    2271 KB |       4 MB |      21 GB |      21 GB |
|---------------------------------------------------------------------------|
| Allocations           |    1150    |    1195    |  220511    |  219361    |
|       from large pool |     344    |     363    |  118388    |  118044    |
|       from small pool |     806    |     842    |  102123    |  101317    |
|---------------------------------------------------------------------------|
| Active allocs         |    1150    |    1195    |  220511    |  219361    |
|       from large pool |     344    |     363    |  118388    |  118044    |
|       from small pool |     806    |     842    |  102123    |  101317    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     162    |     162    |     162    |       0    |
|       from large pool |     135    |     135    |     135    |       0    |
|       from small pool |      27    |      27    |      27    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      69    |      80    |  124656    |  124587    |
|       from large pool |      63    |      71    |   81797    |   81734    |
|       from small pool |       6    |      15    |   42859    |   42853    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

GFPGAN

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4647 MB |    7009 MB |    2134 GB |    2129 GB |
|       from large pool |    4507 MB |    6836 MB |    2116 GB |    2112 GB |
|       from small pool |     139 MB |     178 MB |      17 GB |      17 GB |
|---------------------------------------------------------------------------|
| Active memory         |    4647 MB |    7009 MB |    2134 GB |    2129 GB |
|       from large pool |    4507 MB |    6836 MB |    2116 GB |    2112 GB |
|       from small pool |     139 MB |     178 MB |      17 GB |      17 GB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    7944 MB |    7944 MB |    7944 MB |       0 B  |
|       from large pool |    7762 MB |    7762 MB |    7762 MB |       0 B  |
|       from small pool |     182 MB |     182 MB |     182 MB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |  158609 KB |    1524 MB |    2338 GB |    2338 GB |
|       from large pool |  151678 KB |    1524 MB |    2318 GB |    2318 GB |
|       from small pool |    6931 KB |      30 MB |      20 GB |      20 GB |
|---------------------------------------------------------------------------|
| Allocations           |    2836    |    3208    |  234750    |  231914    |
|       from large pool |     455    |     564    |  126294    |  125839    |
|       from small pool |    2381    |    2659    |  108456    |  106075    |
|---------------------------------------------------------------------------|
| Active allocs         |    2836    |    3208    |  234750    |  231914    |
|       from large pool |     455    |     564    |  126294    |  125839    |
|       from small pool |    2381    |    2659    |  108456    |  106075    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     244    |     244    |     244    |       0    |
|       from large pool |     153    |     153    |     153    |       0    |
|       from small pool |      91    |      91    |      91    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      77    |     108    |  113366    |  113289    |
|       from large pool |      67    |      71    |   74087    |   74020    |
|       from small pool |      10    |      50    |   39279    |   39269    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

[Oceanswave]

Adding torch.cuda.empty_cache() after image processing is complete shows some promising results:

No-GFPGAN

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4085 MB |    6199 MB |    2079 GB |    2075 GB |
|       from large pool |    4039 MB |    6152 MB |    2063 GB |    2059 GB |
|       from small pool |      45 MB |      52 MB |      15 GB |      15 GB |
|---------------------------------------------------------------------------|
| Active memory         |    4085 MB |    6199 MB |    2079 GB |    2075 GB |
|       from large pool |    4039 MB |    6152 MB |    2063 GB |    2059 GB |
|       from small pool |      45 MB |      52 MB |      15 GB |      15 GB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    4196 MB |    7348 MB |   10500 MB |    6304 MB |
|       from large pool |    4148 MB |    7294 MB |   10440 MB |    6292 MB |
|       from small pool |      48 MB |      54 MB |      60 MB |      12 MB |
|---------------------------------------------------------------------------|
| Non-releasable memory |  113503 KB |    1518 MB |    3474 GB |    3474 GB |
|       from large pool |  111232 KB |    1516 MB |    3452 GB |    3452 GB |
|       from small pool |    2271 KB |       4 MB |      21 GB |      21 GB |
|---------------------------------------------------------------------------|
| Allocations           |    1150    |    1195    |  220511    |  219361    |
|       from large pool |     344    |     363    |  118388    |  118044    |
|       from small pool |     806    |     842    |  102123    |  101317    |
|---------------------------------------------------------------------------|
| Active allocs         |    1150    |    1195    |  220511    |  219361    |
|       from large pool |     344    |     363    |  118388    |  118044    |
|       from small pool |     806    |     842    |  102123    |  101317    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     150    |     162    |     174    |      24    |
|       from large pool |     126    |     135    |     144    |      18    |
|       from small pool |      24    |      27    |      30    |       6    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      69    |      80    |  124674    |  124605    |
|       from large pool |      63    |      71    |   81815    |   81752    |
|       from small pool |       6    |      15    |   42859    |   42853    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

GFPGAN:

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    4647 MB |    7009 MB |    2134 GB |    2129 GB |
|       from large pool |    4507 MB |    6836 MB |    2116 GB |    2112 GB |
|       from small pool |     139 MB |     178 MB |      17 GB |      17 GB |
|---------------------------------------------------------------------------|
| Active memory         |    4647 MB |    7009 MB |    2134 GB |    2129 GB |
|       from large pool |    4507 MB |    6836 MB |    2116 GB |    2112 GB |
|       from small pool |     139 MB |     178 MB |      17 GB |      17 GB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |    5366 MB |    7938 MB |    9994 MB |    4628 MB |
|       from large pool |    5192 MB |    7762 MB |    9810 MB |    4618 MB |
|       from small pool |     174 MB |     182 MB |     184 MB |      10 MB |
|---------------------------------------------------------------------------|
| Non-releasable memory |  158609 KB |    1524 MB |    2213 GB |    2213 GB |
|       from large pool |  151678 KB |    1524 MB |    2192 GB |    2192 GB |
|       from small pool |    6931 KB |      30 MB |      20 GB |      20 GB |
|---------------------------------------------------------------------------|
| Allocations           |    2836    |    3208    |  234750    |  231914    |
|       from large pool |     455    |     564    |  126294    |  125839    |
|       from small pool |    2381    |    2659    |  108456    |  106075    |
|---------------------------------------------------------------------------|
| Active allocs         |    2836    |    3208    |  234750    |  231914    |
|       from large pool |     455    |     564    |  126294    |  125839    |
|       from small pool |    2381    |    2659    |  108456    |  106075    |
|---------------------------------------------------------------------------|
| GPU reserved segments |     236    |     244    |     248    |      12    |
|       from large pool |     149    |     153    |     156    |       7    |
|       from small pool |      87    |      91    |      92    |       5    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      77    |     104    |  113111    |  113034    |
|       from large pool |      67    |      71    |   73837    |   73770    |
|       from small pool |      10    |      44    |   39274    |   39264    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

[Oceanswave]

Reported timings:

-n2:

Before Cache Clear:
Non-GFPGAN: 2 images generated in 12.19s
GFPGAN: 2 images generated in 15.10s

After Cache Clear:
Non-GFPGAN: 2 images generated in 12.29s
GFPGAN: 2 images generated in 14.64s

-n10:
Before Cache Clear:
Non-GFPGAN: 10 images generated in 52.17s
GFPGAN: 10 images generated in 54.34s

After Cache Clear:
Non-GFPGAN: 10 images generated in 54.72s
GFPGAN 10 images generated in 55.07s

[smoke2007]

Does it no longer OOM for -n2,3,4,5,... ?

Was it a lot of code changes?

[Oceanswave]

@smoke2007 Are you able to test PR #122 and see if it helps with OOM? Much appreciated

[Oceanswave]

Really small changes - if you could test it that would be great as I don't get OOM in my circumstance with testing up to 300 iterations through gfpgan on a RTX3080 w/24gb

[lstein]

What tool do you use to get those nice resource utilization summary tables? Is it some switch you pass to nvidia-smi? I'd like to benchmark changes to the CUDA code.

My problem with debugging OOM errors is that the only GPUs I have access to are the hefty machines on my university's cluster, which come in 16G or 32G flavors. (I know, play a small violin for my travails...)

[Oceanswave]

print(torch.cuda.memory_summary)

sounds like you're in an awful predicament -- I'm sure one or two of the community here will be happy to help relieve you of your troubles

[Oceanswave]

Since you're so resource poor, could you jam up a sweet self-hosted github action runner that runs a set of regression tests on PR that has access to a GPU? (That sounds too much like work even as I type it)

[lstein]

Which is the key VRAM utilisation column? Is it Peak Usage or Total Allocated?

[DarkStarSword]

I may not be smoke, but this PR did not solve the issue for me - 1st generation runs fine, subsequent generations in -n batch hits OOM (provided there was a face present in the 1st generation).

Hitting up and re-running the same prompt will still do the 1st generation fine, so the VRAM must get cleared at some point.

[smoke2007]

No it didn't solve it entirely either for me, situationally it sometimes continues, but it doesn't seem to help all the time