realesrgan on M1/M2 Macs

[bakkot]

I moved this comment by @eduardoarinopelegrin from here. Original comment follows.

---

Kind of unrelated, but also kind of related (b/c it involves simplet2ti.py).
Has anyone got realesrgan working on M1/M2 Macs with this repo? If not, are there any plans or works in progress?

I just got it to work with the realesrgan-ncnn-vulkan executable + updating the appropriate parts of simplet2ti.py. So if there were no plans to implement / not in the near future, maybe you could include this hack for us M1/M2 Mac users to get it to work.

Steps are:

1. Download the MacOS executable from https://github.com/xinntao/Real-ESRGAN/releases
2. Unzip it (you'll get realesrgan-ncnn-vulkan-20220424-macos) and move realesrgan-ncnn-vulkan inside stable-diffusion (this project folder). Move the Real-ESRGAN model files from realesrgan-ncnn-vulkan-20220424-macos/models into stable-diffusion/models as well.
3. Run chmod u+x realesrgan-ncnn-vulkan to allow it to be run.
4. Update simplet2ti.py to use the executable if self.device.type == 'mps'
   The changes are:

import subprocess

Add a function to do the upscaling

def esrgan_resize(self, input):
    input.save('esrgan_in.png')
    input.save('esrgan_out.png')
    input.close()
    try:
        subprocess.run(
            ['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png'],
            stdout=subprocess.PIPE
        ).stdout.decode('utf-8')
        output = Image.open('esrgan_out.png').convert('RGBA')
        return output
    except Exception as e:
        print('ESRGAN resize failed.')
        print(e)
        quit()

Modify this part to call the function

if upscale is not None:
                                if self.device.type == 'mps':
                                    image = self.esrgan_resize(image)
                                else:
                                    # current code to upscale

This is the full file. This may be more appropriate as a pull request, but since I've never done one (so far!) and simplet2ti.py is being modified (from the version I implemented my changes on), maybe it's easier to add it to your version if you see it fit.
simplet2i.py.zip

Usage:
python3 ./scripts/dream.py
dream > Anubis the Ancient Egyptian God of Death riding a motorbike in Grand Theft Auto V cover, with palm trees in the background, cover art by Stephen Bliss, artstation, high quality -U 4 -m ddim -S 1469565

Obtained/ expected result (4x upscale):

Usage stats:
1 image(s) generated in 32.73s
Outputs:
outputs/img-samples/000144.1469565.png: "Anubis the Ancient Egyptian God of Death riding a motorbike in Grand Theft Auto V cover, with palm trees in the background, cover art by Stephen Bliss, artstation, high quality" -s50 -W512 -H512 -C7.5 -Addim -U 4.0 -F -S1469565

[Any-Winter-4079]

Update to simply say it works with the web version as well.
However, the following part

is restricted by

if (!config.gfpgan_model_exists) {
        document.querySelector("#gfpgan").style.display = 'none';
    }

from static/dream_web/index.js

In my computer this evaluates to false (and doesn't hide it). However, if someone doesn't have it and config.gfpgan_model_exists evaluates to false, it will hide the option to upscale.
Since it's more related to faces and not to simply upscaling, maybe it should be modified as well (?)

[magnusviri]

I absolutely plan on getting this to work because I want it! :D I just haven't had time to work on it.

[thg-muc]

Thank you, works like a charm !

I do have a question though... for me it doesn't matter if i have -U 4 or -U 2, it always performs a 4 time upscale, is it working for you ?

Also, in case somebody has issues with mac security settings with the realesrgan file, i want to recommend to following article to resolve this issue: https://macpaw.com/how-to/allow-apps-anywhere

BR

[Any-Winter-4079]

@thg-muc Thanks for catching that! I forgot about passing the option to realesrgan-ncnn-vulkan. These are the changes to get it to work.
- def esrgan_resize(self, input):
+ def esrgan_resize(self, input, scale):

- image = self.esrgan_resize(image)
+ image = self.esrgan_resize(image, str(upscale[0]))

- subprocess.run(
               ['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png'],
               stdout=subprocess.PIPE
           ).stdout.decode('utf-8')

+ subprocess.run(
               ['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png', '-s', scale],
               stdout=subprocess.PIPE
           ).stdout.decode('utf-8')

For simplicity, here's the updated file
simplet2i.py.zip

[lstein]

If subprocess.run() works similarly on Windows and Linux, then this could be a whole framework for launching post-processing jobs, including ones that aren't native python. This would have the advantage of being both extensible and insulating us from changes in the GFPGAN package.

I see that there are ESRGAN ncnn-vulkan releases for Mac, Ubuntu and Windows. Is there something similar for GFPGAN, or do users have to go through installation from source?

Another question. The subprocess call is using a relative path to the executable. Does run() search the PATH, and is this portable between windows and mac/linux?

[Any-Winter-4079]

I assume subprocess.run() will work very similarly, but I don't have a Windows or Linux machine to test it.

No executables for GFPGAN in Releases https://github.com/TencentARC/GFPGAN/releases
Maybe it could be built, but not readily available.

About subprocess.run() and PATH, not sure. Someone smarter will know :=D
What I will add is I found a problem trying to run realesrgan-ncnn-vulkan from a different directory than where the executable is (passing relative path), and searching the Issues sections, I found the following comment from @VaslD on xinntao/Real-ESRGAN#379

This problem is because realesrgan-ncnn-vulkan tries to read the pretrained model in the working path ($PWD, %CD%), while the usual practice should be in the standard path (platform standard software installation path, user data path, or the bottom line to the executable). file path) to read the data provided with the application installation.

A temporary solution is to run realesrgan-ncnn-vulkan with the full path to the model, using the -m parameter:

/usr/local/RealESRGAN/realesrgan-ncnn-vulkan -m /usr/local/RealESRGAN/models -i ... -o ...
The full path of the model should be spliced ​​with /models in the folder where realesrgan-ncnn-vulkan is located by default. If you put realesrgan-ncnn-vulkan in $PATH, use which under UNIX and where under Windows to get the realesrgan-ncnn-vulkan executable, then get its parent folder and splicing /models as the -m parameter. Can.

Edit: The alternative solution is to cd to the folder where realesrgan-ncnn-vulkan is located as upstairs (because cd has changed the working path), but it is recommended to use pushd instead of cd. After running, popd can return to the original terminal or script. working path.

Also, authors are advised to handle and report exceptions kindly. If the model is not found (models is missing, or -m, -n parameters are wrong), output debugging information and exit with error code instead of triggering [1] 3515 segmentation fault realesrgan-ncnn-vulkan. Otherwise it looks like a careless programming error and doesn't let the caller know that the parameter is wrong.

[glonlas]

On ['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png']
As an optimization too you can add also the param -n to select the model to use.

You will have:

• realesrgan-x4plus <-- Works very well
• realesrgan-x4plus-anime <-- a bit better than x4plus when dealing with manga/drawing
• realesrnet-x4plus <-- slower on my mac
• realesr-animevideov3

[glonlas]

In addition, adding realesrgan in the root folder of this project is not the best.
Hence you can move it to a sibling folder and then send subprocess.run to use that folder with the parameter cwd="../realesrgan"

ex:

            subprocess.run(
                ['./realesrgan-ncnn-vulkan', '-i', 'esrgan_in.png', '-o', 'esrgan_out.png', '-s', scale],
                stdout=subprocess.PIPE,
                cwd="../realesrgan"
            ).stdout.decode('utf-8')

[glonlas]

Here an implementation that works on my side.

1. Download the two files:

• gfpgan_tools.py.patch
• simplet2i.py.patch

You can download with

wget https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1-Install/blob/main/patches/gfpgan_tools.py.patch
wget https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1-Install/blob/main/patches/simplet2i.py.patch

2. Apply the patch with

git apply gfpgan_tools.py.patch
git apply simplet2i.py.patch

Note: I will do a PR tomorrow, this will be cleaner.

[Any-Winter-4079]

Ok, so I tried your code, and it runs
"superman dancing with a panda bear" -s50 -W512 -H512 -C7.5 -Aplms -G0.4 -U 2.0 0.6 -S2516680777
, but something happens with GPFGAN or RealESRGAN
I get this image before

and after, it transforms it into

[Any-Winter-4079]

Face restoration alone (without upscaling) seems to fail as well. Are you using an Intel Mac? I'm on M1.

dream> "superman dancing with a panda bear" -s50 -W512 -H512 -C7.5 -Aplms -G0.4 -S2516680778
Generating:   0%|                                                                                                                                                                             | 0/1 [00:00<?, ?it/s] DEBUG: seed at make_image() invocation time =2516680778
PLMS Sampler: 100%|| 50/50 [00:32<00:00,  1.54it/s]
Generating: 100%|| 1/1 [00:33<00:00, 33.07s/it]
>> GFPGAN - Restoring Faces: superman dancing with a panda bear : seed:2516680778
[W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.
Intel MKL FATAL ERROR: This system does not meet the minimum requirements for use of the Intel(R) Math Kernel Library.
The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions.
The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions.
The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.

[glonlas]

Are you using an Intel Mac? I'm on M1.
Yes I am on M1 as well (2020 13" M1).

I did not get this issue at all. Are you on Main or Development branch?
I am using the main branch from Sep 4, 2022.

[glonlas]

The issue is certainly because my script forces '-n', 'realesrgan-x4plus' which is made for x4 scaling. I need to make it take care of the -U value you are passing.

I could not reproduce the issue, but I will not be surprised it is the root cause for this issue.

[Any-Winter-4079]

I'm using the main branch.

[Any-Winter-4079]

Yes, 4x works.
Okay, so a hack to get 2x to work, is to remove '-n', 'realesrgan-x4plus'
I guess we could remove it -and use the default- unless a specific model is passed.

The default seems to be realesr-animevideov3 (not sure which is better, vs. realesrgan-x4plus)

-n model-name model name (default=realesr-animevideov3, can be realesr-animevideov3 | realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)