CLI: Show Progress / Save Progress / Generate Progress Video

[blessedcoolant]

This is a PR that brings progress features to the CLI. This PR incorporates #623 by @kevinschaul -- Added him to Contributors list.

---

Features

• Show Progress
• Save Progress
• Generate Progress Video

Example Usage:

<prompt> -s 40 -G 1 -show_progress 1 3 -save_progress 1 vo

This command will run the prompt and display a visual preview of the process. The final output will be displayed for 3 seconds before it closes automatically. The prompt has post processing of face restoration. The restoration will take place after the preview closes. Once the restoration is completed, the preview once again loads up to show you the final new updated image for another 3 seconds.

When the process is done, you will find a video of the generation process along with your generated image. More documentation below and in the docs.

---

Show Progress

Provides a visual preview of the image generation process.

-show_progress <step_count: int> <duration: float>

• step_count: The number of steps between each progress update. Default: 5

• duration: The duration (in seconds) for how long you want the final image to be displayed before
  the preview closes automatically. Default: 2

• Enter duration as 0 to keep it open forever until user presses a key. Note that this will block the code
  from running further until user input.

If you have post processing options, the preview will close after image generation and reopen again
with the updated changes.

---

Save Progress & Make Video

Allows you to save the intermediate steps during the image generation process and make a video out
of it.

-save_progress <step_count: int default=5> <video_options: v | vo default: None>

• step_count: The number of steps between each intermediate image saved. When no value is given,
  it defaults to 5.
• video_options: Allows you to generate a video from the intermediate images. Takes two options:
  v (Video) or vo (Video Only)

Usage

-save_progress: Saves intermediate frames every 5 steps. No video generation.

-save_progress 3: Saves intermediate frames every 3 steps. No video generation.

-save_progress 3 v: Saves intermediate frames every 3 steps. Also generates a video from the
frames at the end.

-save_progress 3 vo: Does not save intermediate frames but generates a video of the process every
3 steps.

-show progress 3 -save_progress 3 vo: Shows a preview of the generation process updating every 3
seconds while also saving a video of the same.

---

Performance

I've done testing on various combinations. No issues so far. Fixed all I could find. Feel free to test and report back to me. Thank you.

• There is no obvious memory consumptions changes from any of these features. I don't expect them to be.
• The inference speed however is another story. Because images are being generated, anytime they are loaded to render as preview or save for progress, it slows down the overall inference time a tad bit. On most 8GB systems, this isn't a huge loss given you can now preview and gen videos while the whole process happens. If you're on a lower end system, you can pass a higher step value to keep the inference speed high. Unfortunately there's no way to dodge it == More data == slower. Simple as that.

[SypayV]

Hi,

I just tested this feature, I was looking forward to that, thank you both of you for putting in the efforts.

I noticed that aborting the generation will not increment the image index, causing the next generation to replace the pictures in the intermediate folder. From the behaviour of the script when generating a single image, i was expecting for it to not remove my scraps ... even though now I realise it is not something that was directly designed originally.

Also, from reading your description, I was expecting the script to post-process all intermediates pictures as well.

Thank you again for implementing that, you are opening the door for other cool features to get in.

[blessedcoolant]

I noticed that aborting the generation will not increment the image index, causing the next generation to replace the pictures in the intermediate folder. From the behaviour of the script when generating a single image, i was expecting for it to not remove my scraps ... even though now I realise it is not something that was directly designed originally.

The issue here is that the step_callback does not provide a seed number unlike image_callback. So as a result, I cannot set the file names to have the seed (which would make them unique) unless the user explicitly passes -S to the prompt. That's why in some cases like what you tested, the end up overwriting.

I am trying to figure a way to either extract the seed somehow or try to find another way to uniquely name the intermediate files. If I think of something, I'll update the PR or do another one if this gets merged by then.

Also, from reading your description, I was expecting the script to post-process all intermediates pictures as well.

Nope. Makes no sense to post process images that are mostly noise. GFPGAN wont work on most of them and ESRGAN serves no purpose. No reason at all to enlarge intermediate images.

[mh-dm]

Just tried it out. Like

brown corgi, clean background, flat design -s20 -Addim -S1 -show_progress 1
brown corgi, clean background, flat design -s20 -Addim -S1 -show_progress 5

Both times it goes from ~2.0it/s to ~1.3s/it and updates every step, instead of every 5 for the second one. Tested at 837d00b

EDIT: I thought show_progress would show the progress every 5 steps. Shows that it's easy to confuse. Also, I would want the progress to be computed only every few frames instead of every frame so that it doesn't affect speed much.

Also -show_progress 0 does nothing, which doesn't match documentation.

[mh-dm]

Now I tried:

brown corgi, clean background, flat design -s20 -Addim -S1 -show_progress 1
brown corgi, clean background, flat design -s20 -Addim -S1 -show_progress 5

First ~1.4s/it, second ~1.4it/s.
Second seems to save at the end of iterations 1, 6, 11, 16 and names them 000011.1.01.png through 04.png.
I would expect it would save at the end of iterations 5, 10, 15 instead (not 20 cause that's the end) and name them based on the iteration (i.e. .05.png, .10.png, .15.png).

[blessedcoolant]

@mh-dm Thanks for testing. I made some updates.

Both times it goes from ~2.0it/s to ~1.3s/it and updates every step, instead of every 5 for the second one.
EDIT: I thought show_progress would show the progress every 5 steps. Shows that it's easy to confuse. Also, I would want the progress to be computed only every few frames instead of every frame so that it doesn't affect speed much.

The value on show progress was not a step_count but actually the duration to keep it open at the end. But I realize it is far more intuitive and better to let the user control that.

So I've updated the settings. show_progress now takes two values

-show_progress <step_count: int default: 5> <duration: float default:2>

Usage:

-show_progress 6 3 will update every 6 steps and display final image for 3 seconds.

If both show_progress and save_progress` are given, the step_count of save_progress takes precedence because that involves the more intensive task and I feel it's better to maintain parity with that as the saving in the background goes on.

Second seems to save at the end of iterations 1, 6, 11, 16 and names them 000011.1.01.png through 04.png.

This should be fixed now.

[mh-dm]

If both show_progress and save_progress` are given, the step_count of save_progress takes precedence because that involves the more intensive task and I feel it's better to maintain parity with that as the saving in the background goes on.

That's quite a non-intuitive quirk. I would say to raise an exception that they have to be the same.

This should be fixed now.

The first image with -s20 --save_progress 5 still looks like the output after 1 iteration.

[blessedcoolant]

That's quite a non-intuitive quirk. I would say to raise an exception that they have to be the same.

I don't see a point in raising an exception here saying they have to be the same when we can do it ourselves. Why raise an exception for the user to do something that we are doing anyway? Just an extra unnecessary step.

1. The preview is more of a visual bonus while the saving of files is an actual tangible result. At any given time, I think the tangible output should take precedence.
2. In the bigger picture, this makes almost no difference. Because barely ever does one want to preview at different rate and save at a different rate. In a case like that, having parity between what is happening on the backend and what is being presented to the viewer makes more sense.
3. It's logged in the docs anyway.

This should be fixed now.

The first image with -s20 --save_progress 5 still looks like the output after 1 iteration.

That's just how the iterations are I think. If you log them all, you'll see that the noise extrapolates based on the number of steps.

Typing -s 8 converges euler_a so you'd presume that when you do -s 100 on a euler_a, you'll get a refined result early on, but this is not the case.

Either way, the visual content of the sample images themselves is out of the scope of this PR.

[mh-dm]

The first image with -s20 --save_progress 5 still looks like the output after 1 iteration.

That's just how the iterations are I think. If you log them all, you'll see that the noise extrapolates based on the number of steps.

Typing -s 8 converges euler_a so you'd presume that when you do -s 100 on a euler_a, you'll get a refined result early on, but this is not the case.

Either way, the visual content of the sample images themselves is out of the scope of this PR.

Let me restate for a bit more clarity:
The first image obtained with:
-s20 --save_progress 5
and with
-s20 --save_progress 1
is the same. With the former I would expect to get the 5th image I would get with the latter.

[mh-dm]

I don't see a point in raising an exception here saying they have to be the same when we can do it ourselves. Why raise an exception for the user to do something that we are doing anyway? Just an extra unnecessary step.

For the exact same reason that if the user types in -f2 we raise instead of 'fixing' it for them. We don't know exactly which number they prefer, is it the save_progress or the show_progress one so we let them know early to fix it.

The fun alternative would be to allow different numbers, i.e. solve a real-life version of the fizzbuzz interview question.

[blessedcoolant]

Let me restate for a bit more clarity:
The first image obtained with:
-s20 --save_progress 5
and with
-s20 --save_progress 1
is the same. With the former I would expect to get the 5th image I would get with the latter.

That's because the first image is always being saved as an initial reference point. I can bypass this but is that necessary?

[lstein]

This is very neat indeed. For what it’s worth, I’ll let you know how well the preview window works on a firewalled HPC node. I’m going to have to do two layers of ssh X11 forwarding.

There’s also an old feature request to simply pop up the preview window when the most recent image is done. Maybe that can be funded into this?

[blessedcoolant]

There’s also an old feature request to simply pop up the preview window when the most recent image is done. Maybe that can be funded into this?

I added a check where if you set the -show_progress step count to be the same as the number of steps, you get a preview when the image is generated and no other time. Effectively the same as that requested feature.

For example: <promp> -s 20 -show_progress 20 will give a preview only when the image is generated.

When multiple iterations are being generated, the preview windows updates at the end of each generation.

[mh-dm]

That's because the first image is always being saved as an initial reference point. I can bypass this but is that necessary?

An initial reference point for? I wouldn't call it necessary but there is a drawback: the first step result is not that useful (it's always an unrecognizable blurry mess) and the time could be better spent on the Xth step result (for --save_progress X).

[lstein]

The save-intermediate-images and preview are working great. However, when I try to make a movie (with 'v' or 'vo'), I get a single frame of the final image only.

EDIT: This bug only occurs when you are doing post-processing. With -G1 you get a single-frame image. With no face-fixing, you get the full movie.

I note that I had to replace opencv-python with opencv-contrib python in order to get the libgtk2-compiled module on my Ubuntu system. Is this a general requirement?

[mh-dm]

nonlocals make for more complex code, generally good to be avoided. Options:

• you have access to opt so you could pull step_count of that
• you could use functool.partial(image_progress, a=a) and add another arg to image_process
  whatever you think gets you simpler/shorter code with less potential for errors

[blessedcoolant]

The nonlocals were from the initial PR by kevin that added the progress code. It is similar to how its done on the server.py model too. I'm not a fan of it either but I didn't change it for the time being.

[tildebyte]

I'm with mh-dm on this one. nonlocal is ugly (almost as bad as GOTO 😂)

Unless I'm reading this wrong, this could even be a function outside of main() (still in 'dream.py'), with a (rather long) list of parameters.

[blessedcoolant]

The issue here is that image_callback and step_callback are predefined callbacks that I cannot pass down any values to. And unfortunately the functionality of this and the image writer need it to use values from outside the scope of these functions. And I cannot initialize step_index inside the callback either because then it'll get initialized for iteration which does not work.

With how dream.py is currently written, I don't see a way to bypass this. Probably why even lstein used nonlocals in image_writer.

Let me explore some options to see if there's a cleaner way for me to do this.

[mh-dm]

Remove as you already call waitKey in show_progress?

[blessedcoolant]

Not the same. The waitKey in show_progress is to keep the window alive. The waitKey here lasts for a second so the user gets a preview of the final output for atleast a second before it moves on -- added this because when a user is performing multiple iterations, the final preview gets taken off screen too quickly.

[lstein]

This is very neat indeed. For what it’s worth, I’ll let you know how well the preview window works on a firewalled HPC node. I’m going to have to do two layers of ssh X11 forwarding.

Turns out not to be a problem at all to do all the X11 forwarding. Quite fast and interactive.

[lstein]

There needs to be some type checking on the argument you pass to -save_progress or -show_progress. If you pass a non-numeric argument (as I just did with -save-progress vo you get a crash.

[blessedcoolant]

Agreed. I'm out at the moment. Can't work on this for a bit. If you wanna go ahead and add these tweaks in and make a PR here, I don't mind. If not, I'll do it when I'm back home.

[blessedcoolant]

The save-intermediate-images and preview are working great. However, when I try to make a movie (with 'v' or 'vo'), I get a single frame of the final image only.

EDIT: This bug only occurs when you are doing post-processing. With -G1 you get a single-frame image. With no face-fixing, you get the full movie.

I'll test that out and see why that is happening. Thanks for finding the bug.

I note that I had to replace opencv-python with opencv-contrib python in order to get the libgtk2-compiled module on my Ubuntu system. Is this a general requirement?

Not needed on Windows. So I'm guessing it's a Linux thing?

[lstein]

The save-intermediate-images and preview are working great. However, when I try to make a movie (with 'v' or 'vo'), I get a single frame of the final image only.
EDIT: This bug only occurs when you are doing post-processing. With -G1 you get a single-frame image. With no face-fixing, you get the full movie.

I'll test that out and see why that is happening. Thanks for finding the bug.

I note that I had to replace opencv-python with opencv-contrib python in order to get the libgtk2-compiled module on my Ubuntu system. Is this a general requirement?

Not needed on Windows. So I'm guessing it's a Linux thing?

Very possibly. I just did a full reinstall of the environment so I think my Linux environment is consistent with the standard one. I hate doing this, but maybe we need to make a environment-lin.yaml.

[blessedcoolant]

Very possibly. I just did a full reinstall of the environment so I think my Linux environment is consistent with the standard one. I hate doing this, but maybe we need to make a environment-lin.yaml.

Possibly. If it helps with the setup process, then might as well

Also had to ask, argparse doesn't support individual type checks for an argument which takes more than 1 value right? I'll need to do the type checking manually somewhere in the code?

[kevinschaul]

This is great progress! Really nice work.

I'm running into a few bugs with -show_preview on OS X:

1. The window appears deactivated this even after clicking on it:

2. The window does not close even after the call to cv2.destroyAllWindows(). The only way I can find to close the window is to quit the dream.py script.

[kevinschaul]

cv2 on OS X seems ... not great. But the fix suggested here is closing the preview window for me. Add cv2.waitKey(1) immediately after cv2.destroyAllWindows().

[lstein]

@blessedcoolant Just let me know when you think this is ready for review again.

[blessedcoolant]

@blessedcoolant Just let me know when you think this is ready for review again.

Will do. While the feature is fully functional, I've bumped into a couple of issues that I am finding hard to fix. I am exploring solutions for them. This might take a day or two. I'll let you know when this is ready to go.

[lstein]

@blessedcoolant Just let me know when you think this is ready for review again.

Will do. While the feature is fully functional, I've bumped into a couple of issues that I am finding hard to fix. I am exploring solutions for them. This might take a day or two. I'll let you know when this is ready to go.

No problem. I'm going to be at a conference for the next few days and likely won't be able to do code reviews. Please feel free to step in and do the merge/commits if there are any bugs, nice features or other high pri PRs.

[blessedcoolant]

No problem. I'm going to be at a conference for the next few days and likely won't be able to do code reviews. Please feel free to step in and do the merge/commits if there are any bugs, nice features or other high pri PRs.

I'll be a tad bit busy too but I can pop in. Don't worry about it. Enjoy your trip. Have a good time in Barcelona.

[lstein]

Still in draft, is that right?

[blessedcoolant]

Still in draft, is that right?

Yes. I'm exploring alternative solutions. I'll beep you up on this when I have some updates.

[pigubaoza]

Hi, I found is that the intermediates are not upscaled when using the -U argument in the CLI.

[diffubik]

single-dash arguments are supposed to be used with single-letter flags

this file already has plenty of arguments so maybe you should skip the single-dash arguments entirely

[diffubik]

see https://serverfault.com/a/387936 for more info

[lstein]

@blessedcoolant Still a draft? I thought it was working pretty well, aside from a few minor annoyances.

[blessedcoolant]

@blessedcoolant Still a draft? I thought it was working pretty well, aside from a few minor annoyances.

Yep. I want to leave it as a draft for the time being. The feature works well but there hiccups we know that don't exactly make for a great user experience when encountered. I want to fix them if I can.

Not just that but I also want to look and explore more options than OpenCV for the image display. Especially to avoid the preview window becoming unresponsive (if left open) because it runs on the same thread.

These aren't major issues but are issues I'd like to fix.

I am currently super busy with the WebUI integration and redesign. Once I am done with that, I'll get back to this.

Leaving this open so that if anyone who comes across this and wants to contribute can build off of this rather than start anew.

Let's leave it open for the time being I'd say. Thoughts?

[lstein]

Sounds like a plan! The WebUI is looking fantastic. Have you thought about an integrated eraser tool for inpainting?

[blessedcoolant]

Closing this. There has ben a lot of changes since this PR and with the latent previews, there's a much more optimized way of doing this. I'll work on this at a later period.