[Hub + Examples] Standardize model cards in the training scripts

[sayakpaul]

The model card helper in diffusers does not set either a finetuned_from or base_model metadata.

If we follow the template and start adding the finetuned_from or base_model tag in the metadata portion of the model cards in our training scripts, that will lead to a solid UI improvement on the Hub.

@Wauplin, actually, I need a bit of advice here. I did try to use the template before and found to be a bit non-trivial to use compared to just writing one myself. Could you help with some pointers on how to do this cleanly? You can take this model card from the SDXL DreamBooth LoRA training script as an example.

[Wauplin]

@Wauplin, actually, I need a bit of advice here. I did try to use the template before and found to be a bit non-trivial to use compared to just writing one myself. Could you help with some pointers on how to do this cleanly? You can take this model card from the SDXL DreamBooth LoRA training script as an example.

You can either defined your own template or use the default one. Writing a custom one will be more specialized to your use case (LoRA training for instance) while using the default one helps the user knowing which fields they should provide if they want to create a "good" model card. For the SDXL DreamBooth LoRA example you gave above, you could have:

1. a file lora_template.md:

---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/model-cards
{{ card_data }}
---

# LoRA DreamBooth - {repo_id}

<Gallery />

## Model description 

These are LoRA adaption weights for {{ base_model }}. The weights were trained on '{{ prompt }}' using [DreamBooth](https://dreambooth.github.io/).

LoRA for the text encoder was enabled: {{ train_text_encoder }}.

Special VAE used for training: {{ vae_path }}.

2. a script to populate the template:

from huggingface_hub import ModelCard, ModelCardData


base_model = "sdxl"
prompt = "This is the prompt."
images = [..., ..., ..., ...]
train_text_encoder = True
vae_path = "vae.safetensors"

card = ModelCard.from_template(
    template_path="template.md",
    card_data=ModelCardData(
        license="openrail++",
        base_model=base_model,
        instance_prompt=prompt,
        tags=[
            "stable-diffusion-xl",
            "stable-diffusion-xl-diffusers",
            "text-to-image",
            "diffusers",
            "lora",
        ],
        inference=True,
        widget=[
            {
                "text": prompt,  # TODO: 1 prompt per image?
                "output": {"url": f"image_{i}.png"},
            }
            for i in range(len(images))
        ],
    ),
    train_text_encoder=train_text_encoder,
    vae_path=vae_path,
)

# and then
card.save(local_path)
# or
card.push_to_hub(model_id)

---

The advantage of the template is that you can reuse it, for example if you have a DreamBooth LoRA training script that is not based on SDXL. You would reuse the template but change the tags in ModelCardData.

Note: I took the liberty to use the new <Gallery/> component that uses the widget metadata. It gives you something like this once pushed to the Hub: https://huggingface.co/artificialguybr/3DRedmond-V1

[sayakpaul]

Thank you! @pcuenca WDYT?

[Wauplin]

Also, personal opinion, I would keep a custom model card template as you already did. But also check the default temaplte if some fields could be populated by diffusers (developers, get_started_code, training data,... ?).

[pcuenca]

This looks very nice, thanks @Wauplin! I think it's a good idea to use a template and populate it from the training scripts. I feel we could make it work with just one template for all, don't you think?

[sayakpaul]

Certainly. Let me work on something quick and open this up for discussions. Might need some help. Will see.

[sayakpaul]

Picking it up again. @linoytsaban since you worked on a beautiful model card in #5508, I was wondering if you would like to give this a try. We don't have to do start with all at once, we can just with one and take it from there.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[julien-c]

i think not stale

[linoytsaban]

@sayakpaul oops I missed the tag! I can definitely give it a try

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.