[Tracker] use micro-conditioning for the SDXL trainers

[sayakpaul]

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper here.

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

• DreamBooth SDXL LoRA
• SDXL LCM-LoRA (it needs to be slightly refactored as done in Fix a bug of flip in SDXL training script #6547 by @haofanwang)
• ControlNet SDXL
• T2I Adapter SDXL
• Textual inversion SDXL (same as what's mentioned for SDXL LCM-LoRA above)
• Advanced SDXL trainer

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

[sayakpaul]

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

[charchit7]

Hey @sayakpaul what's the min requirement for GPU while testing for the training?

[sayakpaul]

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

[linoytsaban]

@linoytsaban would you like to take Advanced SDXL trainer?

sure!

[jiqing-feng]

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

Sorry for the delay, I have been quite busy recently; I will do it once I have time.

[ChengsongLu]

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

Can it run locally?

[sayakpaul]

You need to repurpose the code accordingly for that.

[cosmo3769]

Hello @sayakpaul, I would love to work on ControlNet SDXL. Thank you.

[sayakpaul]

Please go ahead.

[jiqing-feng]

Hi @sayakpaul . I think the micro conditioning is already in the example, see conditioning_image_size and conditioning_crop_size, and they are all in the add_time_ids.

BTW, I found a minor error about the resize function, and I already fixed it in #7095

[sayakpaul]

You are right. Thanks for the quick fix.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.