Can it works with sdxl?

[PromeAIpro]

Great, I want to ask if it can work on sdxl? Or even flux？

[Laidawang]

+1

[LituRout]

Yes, it can use sdxl or flux as the underlying generative model instead of stable cascade. Extension to sdxl is straightforward because it has a similar architecture. For flux, you need to adapt the SOC and AFA modules from RB-Modulation into the MM-DiT architecture.

Please let us know if you have any more questions.

[Laidawang]

Are you interested in supporting them? For me, sdxl and flux are closer to the community.

[LituRout]

Yes, it'd be great to support SDXL or Flux. I'm currently working on another project and do not have the bandwidth for this integration. However, if some of you are interested to take the lead, I'd be happy to provide my full support. Since Flux was released after we had completed this project, I'm curious to see what will happen if we integrate Flux in RB-Modulation.

[wuutiing]

Yes, it'd be great to support SDXL or Flux. I'm currently working on another project and do not have the bandwidth for this integration. However, if some of you are interested to take the lead, I'd be happy to provide my full support. Since Flux was released after we had completed this project, I'm curious to see what will happen if we integrate Flux in RB-Modulation.

I am watching into the code trying make it work on xl or flux, and seems that the image feature as K_s \ V_s are derived from an EfficientNetEncoder, which is binded with Stable Cascade. I dont think that encoder will work in xl or flux. Have you any idea about that?

reference code is from : https://github.com/google/RB-Modulation/blob/main/rb-modulation.ipynb

x0_style_forward = models_rbm.effnet(extras.effnet_preprocess(ref_style.to(device)))

[ErwannMillon]

Played with implementing this in flux. Didn't implement the AFA, but the SOC doesn't seem suited to the flux vae.

The stable cascade stage c latent space is extremely compact, and probably much more semantically meaningful as a result.

The flux VAE is comparatively massive (16, 128,128) for a 1024x1024 image. The latent representation is more like a compressed image than a highlevel semantic representation, which is my intuition for why trying to do gradient descent on it to optimize a style loss works badly.