Bulk generation of variations of your favorite image

[morganavr]

This video shows smooth variations of the initial prompt and seed:
https://twitter.com/karpathy/status/1559343616270557184

Source code of this feature is available here:
https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355

Personally I would love to use this feature this way:

dream "an elephant walking on another planet" -S 385838583 -n 1000 -variations

This command will use image from initial seed and then start to change it slightly.
As a result of this command 1000 image files will be created. Then I could scroll through them and enjoy variations of my initially provided image (I provided "image" by specifying -S seed argument)

Without this feature I am forced to manually change prompt in a slow and hacky way by adding random numbers to the prompt and then change them - this makes a new variant of the image that is slightly different from original. But it takes forever to do it manually.
I hope this feature could automate this process.

Hacky way:

dream "an elephant walking on 532 another planet 761" -S 385838583 -n 1
dream "an elephant walking on 531 another planet 762" -S 385838583 -n 1
dream "an elephant walking on 4531 another planet 762" -S 385838583 -n 1

[lstein]

This is a cool idea. So the random numbers in the prompt are just there to stir the pot a bit? This should be way easy to implement under the current framework. Did you want the variations all to be written into the output directory, or should there be a subdirectory named ".variations" ?

[morganavr]

This hack was discovered by users in SD discord - you add a combination of random numbers and letters in random places of your prompt to get variations of the original image but the hack has few problems.

Consider you have this original prompt:

• dream "an elephant walking on 532 another planet 761" -n 1

Then you want to make variations of it, so you use its seed and append one digit to last number:

• dream "an elephant walking on 532 another planet 7615" -n 1 -S 385838583

You could get a slight variation of the original image or you might get pretty much identical one.

Then you continue to change numbers further:

• dream "an elephant walking on 6872 another planet 4215" -n 1 -S 385838583

But it did not work - you get a totally different image, so you try to understand why it did not work (it is safer to add digits at the end of existing number than prepend a digit) and rinse and repeat this hacky process.

But today I accidentally discovered that tweet and was surprised how good author's code is to generate "true" variations of the initial image where any two images are very similar to each other but differ a little. For this Github Issue I was hoping you could integrate author's code to add this powerful "variations engine" to your branch.

I think that "variations engine" will provide more value to users than "hacky" approach with random numbers.

I don't need .variations folder - let a script write variations images into the output directory but to quickly find variation files among others it could be useful to append ".v" string to filename like 000008.2976531757.v.png

[morganavr]

Thinking about it more - even "hack" feature automated would be very interesting to play with :)
I imagine it work this way:

dream "an elephant <<3>> walking on <<5>> another planet <<7>>" -n 100 -S 385838583

<<3>> means that it will be replaced by random 3 digit number.

So the "real" prompt passed to AI will be:

dream "an elephant 387 walking on 06310 another planet 4756102" -n 100 -S 385838583

[morganavr]

Additionally, I think that "hack" variation feature should work in 2 modes:

1. Default mode. In this mode all new images will use seed of the original image. It will result of many versions of original image altered a bit.

Prompt example:

dream "an elephant <<3>> walking on <<5>> another planet <<7>>" -n 100 -S 385838583

2. Sequence (movie) mode. This mode is activated by passing --sequence argument.
   It will use seed of the original image only once. Then it will pass Seed of the last generated image. I expect this mode to be able to generate "movies" - collection of frames that will vary greatly from original image the longer it works but two subsequent images will vary a little.

Prompt example:

dream "an elephant <<3>> walking on <<5>> another planet <<7>>" -n 100 -S 385838583 --sequence

[Oceanswave]

This is interesting - there could be a syntax for token replacement as well. something like ‘’’a ${[old, young, silly, worried]} elephant wearing a ${[gold, red, pink, blue]} ${[dress, tutu, kimono]} <<7>>’’’ to generate all variants and the sequence

[morganavr]

@Oceanswave I believe you are talking about a different feature - prompt generator.
For example, these two prompts differ "only" in color but they produce totally different images (second prompt used Seed from 1st image).

In this thread I am talking about a different feature - creation of "variants" (I don't know how to call it correctly in English).
By variant I mean almost identical image, like 95-99% identical to the source. Look at blue elephant image - a variant would be another image where front leg would be rotated a little different.

Prompts:

elephant wearing green hat -n 1
elephant wearing blue hat -n 1 -S934675406

[lstein]

This is the weird thing about latent spaces. You can type in something like "lkjlafjlasjfaoixuojad" and you'll get a cool image. Just a completely unpredictable one. Folks, I'm going to write some example scripts of how you can do these variations programmatically. Play with them and let me know what the proper artist's workflow is, and then we can design a templating language for the command line together. Lincoln

…

On Tue, Aug 23, 2022 at 2:10 PM morganavr ***@***.***> wrote: @Oceanswave <https://github.com/Oceanswave> I believe you are talking about a different feature - prompt generator. For example, these two prompts differ "only" in color but they produce totally different images. In this thread I am talking about a different feature - creation of "variants" (I don't know how to call it correctly in English). By variant I mean almost identical image, like 95-99% identical to the source. Look at blue elephant image - a variant would be another image where front leg would be rotated a little different. Prompts: elephant wearing green hat -n 1 elephant wearing blue hat -n 1 -S934675406 [image: 000001 934675406] <https://user-images.githubusercontent.com/51041813/186232037-e46d1df2-8afd-4307-bb04-b333d8246ac9.png> [image: 000002 934675406] <https://user-images.githubusercontent.com/51041813/186232045-2047204c-e65b-461a-8e04-d006a8340028.png> — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3EVP2PNC42OVGVJR4X4TV2UHYDANCNFSM57LUTZQA> . You are receiving this because you commented.Message ID: ***@***.***>

-- *Lincoln Stein* Head, Adaptive Oncology, OICR Senior Principal Investigator, OICR Professor, Department of Molecular Genetics, University of Toronto Tel: 416-673-8514 Cell: 416-817-8240 ***@***.*** *E**xecutive Assistant* Michelle Xin Tel: 647-260-7927 ***@***.*** ***@***.***>* *Ontario Institute for Cancer Research* MaRS Centre, 661 University Avenue, Suite 510, Toronto, Ontario, Canada M5G 0A3 @OICR_news <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Foicr_news&data=04%7C01%7CMichelle.Xin%40oicr.on.ca%7C9fa8636ff38b4a60ff5a08d926dd2113%7C9df949f8a6eb419d9caa1f8c83db674f%7C0%7C0%7C637583553462287559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PS9KzggzFoecbbt%2BZQyhkWkQo9D0hHiiujsbP7Idv4s%3D&reserved=0> | www.oicr.on.ca *Collaborate. Translate. Change lives.* This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.

[bakkot]

The technique in the linked gist is neat, if I've understood it correctly: it takes two random seeds, uses those to generate the initial noise matrices, and interpolates between those two matrices.

That suggests a way of doing variations, as long as you have the original seed and prompt: take that seed and generate the same initial matrix, then add a small amount of noise to that (i.e. generate another random matrix, multiply it by a weight < 1, and add it to the original), then re-run the rest of the process with the same prompt using your slightly-different matrix. How much noise you add determines how different the result will be.

It looks like all that's necessary to enable this (for programmatic usage) would be to allow the start_code to be specified.

(Forgive me if I'm using the wrong terms here; I have no idea what I'm doing.)

---

It would be fun if it could read the initial parameters out of the png metadata, so that all you had to do was scripts/re-run.py previous-output.png --strength 0.1 or whatever and it would figure out the rest.

Also, if you have two images generated with the same prompt but different seeds, you can interpolate between them using the same technique.

[bakkot]

I have a script which can variations in #81. It's not ready for general use but if you know a little python and want this feature, give it a try.

[verderog]

This may be of interest here -- but I'm currently working on a plugin/scripting capability on a branch in my fork:

https://github.com/verderog/stable-diffusion/tree/plugin-scripting

See the plugins/ path for info.

Nickel tour:

• Added --plugin plugins.<plugin-name-here> option
• Plugins are simple classes derived from plugins._dreamplugin.DreamPlugin with three methods to automate operations
• I've currently got two examples defined for randomizing input prompts as mentioned in this thread and sweeping the cfg_scale

Operation:

• launch dream.py
• at the dream> prompt, enter your prompt, plugin reference, and plugin input.
• Example#1: dream> an elephant walking on {} another planet {} -S 385838583 -count 10 --plugin plugins.promptrandomizer
• Example#2: dream> dragon --plugin plugins.iteratecfgscale --cs_tart 7.5 --cs_stop 10.0 --cs_step 0.5

Looking for people to test before I submit a PR.

[verderog]

Fork now includes an example of "img2imgchain" plugin where a base image is morphed for a number of generations using a fixed strength setting.

Example: black dragon --count 10 --chain_base outputs/img-samples/000001.3302085031.png --chain_strength 0.5 --plugin plugins.img2imgchain -> starts with the 000001..image, and runs 10 iterations, using the previous iteration's output for a subsequent iteration's input.

[verderog]

@lstein Is there any interest in me submitting my plugin-scripting branch as PR? If so, I can integrate the latest main branch changes into my fork and submit that. (Reference: https://github.com/verderog/stable-diffusion/tree/plugin-scripting)

On a related note, I made a new plugin today named "artistrendering". Inspired by https://old.reddit.com/r/StableDiffusion/comments/x32dif/study_of_500_artists_using_the_same_prompt/, I wrote this plugin that iterates through 442 different artists with a common prompt and seed, changing only the artist name. Includes an option to use a randomly selected subset of artists from the full list.

Example invocation: a beautiful elf princess, ethereal face. Painted by {} --steps 15 --plugin plugins.artistrendering

[tildebyte]

@bakkot, @morganavr; Can we close this as complete based on #277? Or is this a request for something completely different?

@verderog; Your work is really cool, and I think people here would be very interested in taking a look at it, but it's pretty OT for this issue. Please either open a PR with your contributions, or have a look at #155 and see if what you have is what's being asked for there (or can at least get the ball rolling).

Really, you might as well open a PR and see what people think of it 😁

[morganavr]

Closed as completed in #277