refactoring internal tracking of parameters

[bakkot]

This is mostly a heads-up for @lstein @tildebyte etc: I want to refactor the internal tracking of image settings so that there's a single class which holds the information for generating an image (prompt, seed, steps, sampler, etc). It could be parsed from, and serialized to, both structured metadata (with a translation layer for the extra fields) and prompts.

This will touch a lot of code, so I want to make sure no one objects or is doing something which would get disrupted by this before I start.

[tildebyte]

This is a fantastic idea, and one of the first refactorings I did in my personal copy of RHW's CLIP-Guided-diffusion notebook last year (basically I gathered all of the scattered generation parameters into a dict, and used that dict throughout; once generation is done and it's time to write out PNG metadata, it's just a simple iteration)

[lstein]

This sounds extremely useful. I'm trying to complete a refactoring of the image generation code into a more sensible series of modules, and what I've got so far is in branch refactoring-t2i. It's not currently functional, but I'm hoping to get it all working before bedtime. You'll probably want to make your changes against this branch. I'll drop a note here when it is done.

[lstein]

Hi @bakkot. I've finished the radical part of the refactoring, and it can be found in public branch refactoring-simplet2i. The total lines of code have gone down, so it's got to be an improvement. I really like your idea of packing all the generation variables into a dict and just passing that around.

So if you're up to it, please feel free to fork the refactoring branch and work against it. I'm also happy for you to make direct commits into the branch as long as you're being careful.

[SMUsamaShah]

I didn't go through the all comments in RFC. From what i have read, are you proposing to store JSON in the png files? If so, IMHO we should keep the current command like format in the png. It's both readable and copy/pastable from image viewers. It does not need more tooling to first turn it into a copyable command.

[bakkot]

I will also keep the current command format in the png, yes.

[lstein]

I ended up with a straightforward version of a class that gathers together all the image generation parameters when I did an implementation of RFC #266. It's not quite finished, but is contained in draft #587 . I hope this hasn't overlapped with any work you might have started.

A design question: The call signature for prompt2image() is really ugly because of this sort of thing:

   prompt2img(
            iterations       = None,
            steps            = None,
            seed             = None,
            cfg_scale        = None,
            ddim_eta         = None,
            skip_normalize   = False,
            image_callback   = None,
            step_callback    = None,
            width            = None,
            height           = None,
            sampler_name     = None,
            seamless         = False,
            log_tokenization = False,
  ...etc for another 12 lines
  )

I can now replace this with def prompt2image(self,args), where args is the object that gathers up all the image generation options. Because of the way that args is created, I know that it has all the required keys and the right default values, so it is OK for use in the web and CLI scripts. However, it makes it much harder for people who are writing scripts against Generate, because the type checking is gone. Should I leave things be?