Upscaling with img2img and Real-ESRGAN

[1blackbar]

https://github.com/jquesnelle/txt2imghd
This is prettuy great and results are impressive, he splits big upscale from ESRGAN into small chunks and upscales using img2img and same prompt , so this gives best result.
https://raw.githubusercontent.com/jquesnelle/txt2imghd/master/gallery/00091ud.png

More samples https://github.com/jquesnelle/txt2imghd/tree/master/gallery
You think incorporating it into the webui would be feasible ?

[blessedcoolant]

I wanted to add this. I implemented it on my end and functionally it works but I realized the results are not always great. But I guess it's more to do with the limitations of the technique rather than the implementation itself.

I'll refine this and maybe make a PR sometime later today or tomorrow.

[Visual-Synthesizer]

ive been doing this manually with gigapixel, DD, or JAX diffusion depending on the image. good to know that img2img also is very useful!

[1blackbar]

I wanted to add this. I implemented it on my end and functionally it works but I realized the results are not always great. But I guess it's more to do with the limitations of the technique rather than the implementation itself.

I'll refine this and maybe make a PR sometime later today or tomorrow.

That would be super awesome, there so many styles that i m sure it will work amazing on some but not so much on other ones, but having it as option would be fantastic from what ive seen in gallery results, do you have some results somewhere ?
Maybe good pipline about it would be adjustable layer-like blend of - original image - sharpened image with basic 2d sharpen after 4x upscale - img2img result. So blended 3 layers, with 3 sliders.
while AI upscale is good on some styles i found that simple sharpening of pythons filter2d can get good results blended/merged with ai upscale https://www.analyticsvidhya.com/blog/2021/08/sharpening-an-image-using-opencv-library-in-python/#:~:text=To%20sharpen%20an%20image%20in,image%20that%20will%20undergo%20sharpening.

[blessedcoolant]

I wanted to add this. I implemented it on my end and functionally it works but I realized the results are not always great. But I guess it's more to do with the limitations of the technique rather than the implementation itself.
I'll refine this and maybe make a PR sometime later today or tomorrow.

That would be super awesome, there so many styles that i m sure it will work amazing on some but not so much on other ones, but having it as option would be fantastic from what ive seen in gallery results, do you have some results somewhere ? Maybe good pipline about it would be adjustable layer-like blend of - original image - sharpened image with basic 2d sharpen after 4x upscale - img2img result. So blended 3 layers, with 3 sliders. while AI upscale is good on some styles i found that simple sharpening of pythons filter2d can get good results blended/merged with ai upscale https://www.analyticsvidhya.com/blog/2021/08/sharpening-an-image-using-opencv-library-in-python/#:~:text=To%20sharpen%20an%20image%20in,image%20that%20will%20undergo%20sharpening.

I'll work on it in the coming days.

I'm currently working to optimize the GFPGAN patch. That patch already implements ESRGAN to this project. The link you gave also uses ESRGAN but it uses the external CLI for it. Doing both would be redundant. I need to figure out a way to implement this same thing but using the ESRGAN that's already in the project. Not to mention optimizing it. Shouldn't be too hard but need to figure out the kinks.

[Ratinod]

I was able to run "jquesnelle/txt2imghd" in "CompVis/stable-diffusion". (# all references to "watermark / wm").
Yes, the result is not always good. And quite often this method tries to add something to an empty place (artifacts or something from the prompt). But when it works, it works just awesome!
Very lacking interactive command-line :/
I also tried to do it manually. generate image -> upscale -> slicing -> img2img -> image merging... It turns out a lot of manual work, but the result is impressive. The main thing is not to cut the images without overlaps. :)

[blessedcoolant]

I was able to run "jquesnelle/txt2imghd" in "CompVis/stable-diffusion". (# all references to "watermark / wm"). Yes, the result is not always good. And quite often this method tries to add something to an empty place (artifacts or something from the prompt). But when it works, it works just awesome! Very lacking interactive command-line :/ I also tried to do it manually. generate image -> upscale -> slicing -> img2img -> image merging... It turns out a lot of manual work, but the result is impressive. The main thing is not to cut the images without overlaps. :)

The main issue I found with the method is that the prompt needs to be super detailed. If it's trying to fill spaces that empty, it often tries to add the same stuff across all the newly generated images. So you end up with say birds in all the prompt generations instead of them just being in the sky.

It's definitely not for all types of prompts but it works great when it does. I actually did a manual try too before I tried this code. It got much better results b ut that's because I was very careful to get the right kind of effect I needed. But it was a LOT of work.

[Ratinod]

The main issue I found with the method is that the prompt needs to be super detailed. If it's trying to fill spaces that empty, it often tries to add the same stuff across all the newly generated images. So you end up with say birds in all the prompt generations instead of them just being in the sky.

It's definitely not for all types of prompts but it works great when it does. I actually did a manual try too before I tried this code. It got much better results b ut that's because I was very careful to get the right kind of effect I needed. But it was a LOT of work.

I agree. It's not perfect and it takes a long time. But overall, I'm happy with this find. In fact, it turns out to be a universal upscaler! It's almost like DFDNet, but for anything (of course a detailed prompt is required). it's just awesome.

[lstein]

I've done limited testing, but the upscaling works great nearly all the time. The face fixing is hit-and-miss. It works better for large faces than smaller ones.

Lincoln