[Model Card] standardize dreambooth model card by sayakpaul · Pull Request #6729 · huggingface/diffusers

sayakpaul · 2024-01-27T10:30:18Z

What does this PR do?

An attempt to partially fix#5667 by starting to standardize the creation of model card in the train_dreambooth.py script. An immediate follow-up of #6678 as well. @linoytsaban FYI as well.

I have taken the liberty to fix some type annotations as well.

Generated model card: https://huggingface.co/sayakpaul/test-model-card-template-dreambooth

Notebook to test: https://huggingface.co/sayakpaul/test-model-card-template-dreambooth/blob/main/test_dreambooth_model_card.ipynb (hosted on Hub and I demand a gift for using the Hub to host the notebook :v)

HuggingFaceDocBuilderDev · 2024-01-27T10:42:33Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

linoytsaban · 2024-01-29T09:15:02Z

@sayakpaul thanks for the mention! Direct use will be used for a code example showing how to inference with diffusers?

sayakpaul · 2024-01-29T09:27:24Z

No, those will have to be changed by the model developer. We don't automatically generate "direct use" for all model cards. I think it's okay not to and edit them manually. But I don't think it should be a strict requirement. This is because for ControlNet training it's different direct use, for DreamBooth training, it's different direct use. Hope that make sense.

patrickvonplaten

Thanks for following up with the updates that quickly! Would favor not adding kwargs here to the function signature - apart from this, the PR looks great to me!

Can we update the other example scripts as well?

sayakpaul · 2024-01-29T15:17:36Z

@patrickvonplaten I am not a big fan of kwargs either, but if you see how we're creating the card from the script:

model_card=load_or_create_model_card(
repo_id_or_path=repo_id,
license="creativeml-openrail-m",
base_model=base_model,
instance_prompt=prompt,
model_description=model_description,
inference=True,
)

Should we make everything explicit in the signature? Do we really need to? Or maybe we could just validate with a decorator and keep the kwargs specific to the training script only. I think the second one is a more lenient and cleaner approach.

Once we settle on that, I will update the example scripts where we create model cards. For the ones, that don't, they can be done in a separate PR, preferably by the community.

yiyixuxu

the code looks good to me once @patrickvonplaten 's comments are addressed

Why did we decide to use the default model card template? this https://huggingface.co/sayakpaul/test-model-card-template-dreambooth doesn't look very nice with so many empty fields

sayakpaul · 2024-02-01T07:16:28Z

the code looks good to me once @patrickvonplaten 's comments are addressed

Well my questions need to be addressed first :D #6729 (comment)

Why did we decide to use the default model card template? this https://huggingface.co/sayakpaul/test-model-card-template-dreambooth doesn't look very nice with so many empty fields

It’s the standard one I followed following trainers from other libraries. I think we can iterate on the template in future PRs. Ccing @younesbelkada to check if he has anything to suggest from the TRL world.

yiyixuxu · 2024-02-01T08:09:20Z

@sayakpaul

Well my questions need to be addressed first :D #6729 (comment)

ahhh you're absolutely right!! sorry I misread your code🙈
I'm in favor of making the signature explicit and only adding arguments that are needed. I'm also in favor of a template that only has the minimum number of fields that we need:)

I think we should decide on a template in this PR, though. The goal here is to standardize model cards, and the template is a super important part of it, no?

sayakpaul · 2024-02-01T08:18:53Z

You are right. I will get to the changes.

younesbelkada · 2024-02-01T08:21:54Z

It’s the standard one I followed following trainers from other libraries. I think we can iterate on the template in future PRs. Ccing @younesbelkada to check if he has anything to suggest from the TRL world.

Indeed that's what we use by default in transformers too ! If you prefer you could also set an empty model card instead of using a template

sayakpaul · 2024-02-01T08:23:21Z

If you prefer you could also set an empty model card instead of using a template

Do you have a reference?

sayakpaul · 2024-02-05T05:25:58Z

@yiyixuxu WDYT about the recent changes? :)

Example: https://huggingface.co/sayakpaul/test-model-card-template-dreambooth.

yiyixuxu · 2024-02-05T08:02:19Z

@sayakpaul
I like that:)

sayakpaul · 2024-02-05T08:04:12Z

I am now going to propagate the changes to the rest of the scripts and will ask for a review once done. Thanks!

sayakpaul · 2024-02-05T09:20:56Z

SDXL looks like so: https://huggingface.co/sayakpaul/test-sdxl-lora-dreambooth-model-card.

sayakpaul · 2024-02-05T09:22:30Z

- base_model=str,
+ base_model: str = None,
 train_text_encoder=False,
- instance_prompt=str,
- validation_prompt=str,
+ instance_prompt=None,
+ validation_prompt=None,


Was having compulsion disorder :3 So, decided to fix these.

I would even update to this 😄
base_model: Optional[str] =None, train_text_encoder: bool=False, instance_prompt: Optional[str] =None, validation_prompt: Optional[str] =None,

sayakpaul · 2024-02-05T09:31:12Z

@yiyixuxu this is up for another review. I decided to standardize the DreamBooth scripts and the Custom Diffusion script.

Will open the rest for the community. WDYT?

Wauplin

Thanks for working on that @sayakpaul! Looks good to me :) Added a few nit comments + suggestion to add the <Gallery/> component to nicely display examples in the model card.

Wauplin · 2024-02-06T17:46:21Z

- base_model=str,
+ base_model: str = None,
 train_text_encoder=False,
- instance_prompt=str,
- validation_prompt=str,
+ instance_prompt=None,
+ validation_prompt=None,


I would even update to this 😄
base_model: Optional[str] =None, train_text_encoder: bool=False, instance_prompt: Optional[str] =None, validation_prompt: Optional[str] =None,

Wauplin · 2024-02-06T17:58:32Z

+ widget=widget_str,
+ ),
+ template_path=MODEL_CARD_TEMPLATE_PATH,
+ model_description=model_description,


If widget is populated, I would automatically add a </Gallery> tag to the model card description like done in https://huggingface.co/Pclanglais/Mickey-1928/blob/main/README.md?code=true. It will nicely render example images in a gallery (see here) and you wouldn't have to manually build a img_str in your notebook example. What do you think? :)

(maybe only if widget is not none and "<gallery>/" not in model_description)

That means the existing save_model_card will have to be rewritten a bit to support compatibility with </Gallery>. I propose not to do that in this PR as it differs in scope.

Fine for me!

yiyixuxu

thank you:)

yiyixuxu · 2024-02-06T20:35:21Z

SDXL looks like so: https://huggingface.co/sayakpaul/test-sdxl-lora-dreambooth-model-card.

these [to-do] fields are intended to be filled manually by users, right?

Wauplin

Thanks @sayakpaul! Looks good 🔥

* feat: standarize model card creation for dreambooth training. * correct 'inference * remove comments. * take component out of kwargs * style * add: card template to have a leaner description. * widget support. * propagate changes to train_dreambooth_lora * propagate changes to custom diffusion * make widget properly type-annotated

sayakpaul added 2 commits January 27, 2024 15:51

feat: standarize model card creation for dreambooth training.
fe67424

correct 'inference
8c84d91

sayakpaul requested review from Wauplin and patrickvonplaten January 27, 2024 10:30

sayakpaul added 3 commits January 27, 2024 16:00

remove comments.
9bc1fbc

take component out of kwargs
f463fe2

style
dc9afd3

Merge branch 'main' into standarize-mcard-dreambooth
108109c

patrickvonplaten reviewed Jan 29, 2024
View reviewed changes

Comment threadsrc/diffusers/utils/hub_utils.py Outdated

patrickvonplaten reviewed Jan 29, 2024
View reviewed changes

Comment threadsrc/diffusers/utils/hub_utils.py Outdated

patrickvonplaten reviewed Jan 29, 2024
View reviewed changes

Comment threadsrc/diffusers/utils/hub_utils.py Outdated

patrickvonplaten reviewed Jan 29, 2024
View reviewed changes

sayakpaul requested a review from yiyixuxu January 31, 2024 09:15

Merge branch 'main' into standarize-mcard-dreambooth
458c37a

yiyixuxu reviewed Feb 1, 2024
View reviewed changes

add: card template to have a leaner description.
52d8131

Merge branch 'main' into standarize-mcard-dreambooth
07de49d

widget support.
2f647dd

sayakpaul commented Feb 5, 2024
View reviewed changes

sayakpaul added 2 commits February 5, 2024 14:56

propagate changes to train_dreambooth_lora
cc1c73e

propagate changes to custom diffusion
2650fed

sayakpaul requested review from patrickvonplaten and yiyixuxu February 5, 2024 09:31

Wauplin reviewed Feb 6, 2024
View reviewed changes

yiyixuxu approved these changes Feb 6, 2024
View reviewed changes

make widget properly type-annotated
2c0a3cf

Wauplin approved these changes Feb 7, 2024
View reviewed changes

Merge branch 'main' into standarize-mcard-dreambooth
2f4e9b2

sayakpaul mentioned this pull request Feb 7, 2024
[Tracker] use the new model card utilities for saving model cards from the training script #6891
Closed
11 tasks

sayakpaul merged commit 76696dc into mainFeb 7, 2024

sayakpaul deleted the standarize-mcard-dreambooth branch February 7, 2024 09:37

bamps53 mentioned this pull request Feb 7, 2024
fix: keyword argument mismatch #6895
Merged
6 tasks

cosmo3769 mentioned this pull request Feb 13, 2024
[Model Card] standardize T2I Adapter Sdxl model card #6947
Merged
6 tasks

Uh oh!

Conversation

sayakpaul commented Jan 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Jan 27, 2024

Uh oh!

linoytsaban commented Jan 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Jan 29, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jan 29, 2024

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Feb 1, 2024

Uh oh!

yiyixuxu commented Feb 1, 2024

Uh oh!

sayakpaul commented Feb 1, 2024

Uh oh!

younesbelkada commented Feb 1, 2024

Uh oh!

sayakpaul commented Feb 1, 2024

Uh oh!

sayakpaul commented Feb 5, 2024

Uh oh!

yiyixuxu commented Feb 5, 2024

Uh oh!

sayakpaul commented Feb 5, 2024

Uh oh!

sayakpaul commented Feb 5, 2024

Uh oh!

sayakpaulFeb 5, 2024

Choose a reason for hiding this comment

Uh oh!

WauplinFeb 6, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Feb 5, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

WauplinFeb 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WauplinFeb 6, 2024

Choose a reason for hiding this comment

Uh oh!

WauplinFeb 6, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaulFeb 7, 2024

Choose a reason for hiding this comment

Uh oh!

WauplinFeb 7, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Feb 6, 2024

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jan 27, 2024•
edited
Loading

linoytsaban commented Jan 29, 2024•
edited
Loading

sayakpaul commented Feb 5, 2024•
edited
Loading