Add IP Adapter training script and update the docs with instructions by AMohamedAakhil · Pull Request #7196 · huggingface/diffusers

AMohamedAakhil · 2024-03-03T19:27:01Z

What does this PR do?

This PR adds the original IP Adapter training scripts, and also updates the documentation with instructions on how to use it.

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

AMohamedAakhil · 2024-03-03T19:28:02Z

#7194

HuggingFaceDocBuilderDev · 2024-03-04T03:08:22Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul · 2024-03-04T03:28:58Z

+#### Usage Example:
+```
+accelerate launch --num_processes 8 --multi_gpu --mixed_precision "fp16" \
+ tutorial_train_ip-adapter.py \
+ --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5/" \
+ --image_encoder_path="{image_encoder_path}" \
+ --data_json_file="{data.json}" \
+ --data_root_path="{image_path}" \
+ --mixed_precision="fp16" \
+ --resolution=512 \
+ --train_batch_size=8 \
+ --dataloader_num_workers=4 \
+ --learning_rate=1e-04 \
+ --weight_decay=0.01 \
+ --output_dir="{output_dir}" \
+ --save_steps=10000
+```


Lets first provide a single-GPU example command and then proceed to multi-GPU.

sayakpaul · 2024-03-04T03:29:35Z

+import torch
+
+# Load the trained model checkpoint
+ckpt = "checkpoint-50000/pytorch_model.bin"


Let's default to serializing in .safetensors please as it's a more secure file-format.

sayakpaul · 2024-03-04T03:31:05Z

+- `map_location="cpu"`: Specifies that the model should be loaded onto the CPU.
+- `image_proj_sd`: Dictionary to store the components related to image projection.
+- `ip_sd`: Dictionary to store the components related to the IP adapter.
+- `"unet"`, `"image_proj_model"`, `"adapter_modules"`: Prefixes indicating components of the model.


But we are not showing actual inference here no? I think we are just showing a part of the process. How to run inference with "ip_adapter.bin" is missing.

sayakpaul · 2024-03-04T03:32:26Z

+ block_id = int(name[len("down_blocks.")])
+ hidden_size = unet.config.block_out_channels[block_id]
+ if cross_attention_dim is None:
+ attn_procs[name] = AttnProcessor()


We should use AttnProcessor2_0() when using PyTorch 2.0 as it's more memory and compute efficient.

It's imported AttnProcessor2_0 as AttnProcessor

sayakpaul · 2024-03-04T03:32:32Z

+ "to_k_ip.weight": unet_sd[layer_name + ".to_k.weight"],
+ "to_v_ip.weight": unet_sd[layer_name + ".to_v.weight"],
+ }
+ attn_procs[name] = IPAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, num_tokens=num_tokens)


Same as above.

Also imported IPAttnProcessor2_0 as IPAttnProcessor

Better would be to first check if it's using PyTorch 2. and then dynamically selecting it.
An example:
diffusers/src/diffusers/models/attention_processor.py
Line 214 in 869bad3
AttnProcessor2_0() ifhasattr(F, "scaled_dot_product_attention") andself.scale_qkelseAttnProcessor()

sayakpaul

Thanks! I think the structure of the training scripts deviate quite a bit from how our official training scripts are written. For now, let's maybe put these under research_projects?

AMohamedAakhil · 2024-03-04T03:38:18Z

Sure!

AMohamedAakhil · 2024-03-04T03:41:06Z

And I'll work on the suggested changes

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

kadirnar · 2024-03-26T22:10:21Z

@AMohamedAakhil ,

Can you give information about the json file? Can you share a sample json file? How should I prepare a dataset?

github-actions · 2024-04-20T15:03:19Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

sayakpaul · 2024-06-30T05:40:06Z

To start, maybe we could add this to research_projects?

sayakpaul · 2024-07-26T05:35:13Z

@AMohamedAakhil we would very much like to add this to research_projects. Can we revive this PR?

AMohamedAakhil · 2024-07-26T05:35:51Z

Sure @sayakpaul

sayakpaul · 2024-07-27T03:51:10Z

Thanks! Please let us know once done!

github-actions · 2024-09-14T15:18:16Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions · 2024-10-12T15:09:00Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

AMohamedAakhiland others added 2 commits March 4, 2024 00:47

docs: add IP Adapter training instructions
10ef761

Delete venv
097f01e

yiyixuxu requested a review from sayakpaul March 4, 2024 01:31

sayakpaul reviewed Mar 4, 2024
View reviewed changes

Comment threadexamples/ip_adapter/tutorial_train_sdxl.py Outdated

sayakpaul reviewed Mar 4, 2024
View reviewed changes

Update examples/ip_adapter/tutorial_train_sdxl.py
d077115
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

AMohamedAakhil mentioned this pull request Mar 8, 2024
Train IP Adapter #7194
Closed

github-actionsBot added the stale Issues that haven't received updates label Apr 20, 2024

yiyixuxu removed the stale Issues that haven't received updates label Apr 22, 2024

github-actionsBot added the stale Issues that haven't received updates label Sep 14, 2024

yiyixuxu removed stale Issues that haven't received updates labels Sep 17, 2024

github-actionsBot added the stale Issues that haven't received updates label Oct 12, 2024

ParagEkbote mentioned this pull request Nov 19, 2024
Move IP Adapter Scripts to research project #9960
Merged
5 tasks

stevhliu closed this in #9960 Nov 19, 2024

Uh oh!

Conversation

AMohamedAakhil commented Mar 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

AMohamedAakhil commented Mar 3, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Mar 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

AMohamedAakhil commented Mar 4, 2024

Uh oh!

AMohamedAakhil commented Mar 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kadirnar commented Mar 26, 2024

Uh oh!

github-actionsBot commented Apr 20, 2024

Uh oh!

sayakpaul commented Jun 30, 2024

Uh oh!

sayakpaul commented Jul 26, 2024

Uh oh!

AMohamedAakhil commented Jul 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Jul 27, 2024

Uh oh!

github-actionsBot commented Sep 14, 2024

Uh oh!

github-actionsBot commented Oct 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AMohamedAakhil commented Mar 3, 2024•
edited
Loading

AMohamedAakhil commented Mar 4, 2024•
edited
Loading

AMohamedAakhil commented Jul 26, 2024•
edited
Loading