Is your feature request related to a problem? Please describe.
Using one of the attention.py changes I'm generating 2560x1024 or larger images. For ad-hoc examples see here: https://sirisian.com/stablediffusion/
Other than the well known 512x512 tiling issues, generating large images often requires variety across the whole image. Current prompts and weights are global and create samey generation when scaled up. My big picture goal is generating wallpapers for fun without requiring postprocessing or multiple steps.
Describe the solution you'd like
There are probably a few solutions, but I'm not sure if all are possible. I'd like to keep this so it works in the console. I have seen the interpolation discussions in like #184, but I'm not familiar enough to say if they're applicable.
Right now you can weight a token with example:0.5, but these are all global changes. The simplest change I can think of is a linear gradient weight change. desert:{type: 'lineargradient', x0: 0, y0: 0, x1: 1, y1: 0, weight: [1, 0] }, forest:{type: 'lineargradient', x0: 0, y0: 0, x1: 1, y1: 0, weight: [0, 1] }. That would go from desert on the left of the image to a forest on the right of the image. This covers a lot of various landscape (desert to forest), weather (rain to snow), or time of day (twilight to bright sky) changes one might do.
Supporting radialgradient might be enough to support localized changes. Like if one wants a cabin in a specific spot or a lake in a general area. {type: 'radialgradient', x: 0.5, y: 0.5, rx: 0.2, ry: 0.2, weight: [1, 0] }.
The more complex positional weights could be handled by a mask with a specific color. river:{ image: 'weights.png' colors: ['#000000', '#ffffff'], weights: [0.5, 1] }. This would function by looking at every pixel and calculating the naive position between the colors (looking the R, G, and B channel differences) and normalizing to the weight range. By using a single image one could encode multiple features. For example: desert: {image: 'weights.png', colors: ['#000000', '#ff0000'], weight: [0, 1] }, forest: {image: 'weights.png', colors: ['#000000', '#00ff00'], weight: [0, 1] }. A color value #880000 would have a weight for desert of 0.53 and for forest a weight of 0.
Describe alternatives you've considered
img2img can help a little bit to introduce variety, but I haven't generated anything large with it that I've been happy with. Perhaps someone else has a workflow that creates large varied images consistently. Generating gradual changes across an image I don't think is possible with it.
In theory inpainting with new prompts works, but this is a very manual process requiring multiple generations. Scripting an inpainting algorithm with prompt weights that change and growing the image I believe might work, but it would take an incredibly long time to get a result. (Large images with a lot of samples already takes 3-16 minutes).
Additional context
I haven't seen any fork that does something like this, so perhaps modifying the sampler like this isn't possible or easy? If it's not then feel free to close this.
Is your feature request related to a problem? Please describe.
Using one of the attention.py changes I'm generating 2560x1024 or larger images. For ad-hoc examples see here: https://sirisian.com/stablediffusion/
Other than the well known 512x512 tiling issues, generating large images often requires variety across the whole image. Current prompts and weights are global and create samey generation when scaled up. My big picture goal is generating wallpapers for fun without requiring postprocessing or multiple steps.
Describe the solution you'd like
There are probably a few solutions, but I'm not sure if all are possible. I'd like to keep this so it works in the console. I have seen the interpolation discussions in like #184, but I'm not familiar enough to say if they're applicable.
Right now you can weight a token with
example:0.5, but these are all global changes. The simplest change I can think of is a linear gradient weight change.desert:{type: 'lineargradient', x0: 0, y0: 0, x1: 1, y1: 0, weight: [1, 0] }, forest:{type: 'lineargradient', x0: 0, y0: 0, x1: 1, y1: 0, weight: [0, 1] }. That would go from desert on the left of the image to a forest on the right of the image. This covers a lot of various landscape (desert to forest), weather (rain to snow), or time of day (twilight to bright sky) changes one might do.Supporting radialgradient might be enough to support localized changes. Like if one wants a cabin in a specific spot or a lake in a general area. {type: 'radialgradient', x: 0.5, y: 0.5, rx: 0.2, ry: 0.2, weight: [1, 0] }.
The more complex positional weights could be handled by a mask with a specific color.
river:{ image: 'weights.png' colors: ['#000000', '#ffffff'], weights: [0.5, 1] }. This would function by looking at every pixel and calculating the naive position between the colors (looking the R, G, and B channel differences) and normalizing to the weight range. By using a single image one could encode multiple features. For example:desert: {image: 'weights.png', colors: ['#000000', '#ff0000'], weight: [0, 1] }, forest: {image: 'weights.png', colors: ['#000000', '#00ff00'], weight: [0, 1] }. A color value#880000would have a weight for desert of 0.53 and for forest a weight of 0.Describe alternatives you've considered
img2img can help a little bit to introduce variety, but I haven't generated anything large with it that I've been happy with. Perhaps someone else has a workflow that creates large varied images consistently. Generating gradual changes across an image I don't think is possible with it.
In theory inpainting with new prompts works, but this is a very manual process requiring multiple generations. Scripting an inpainting algorithm with prompt weights that change and growing the image I believe might work, but it would take an incredibly long time to get a result. (Large images with a lot of samples already takes 3-16 minutes).
Additional context
I haven't seen any fork that does something like this, so perhaps modifying the sampler like this isn't possible or easy? If it's not then feel free to close this.