Skip to content

[PEFT] Adapt example scripts to use PEFT - #5388

Merged
sayakpaul merged 38 commits into
huggingface:mainfrom
younesbelkada:adapt-example-peft
Dec 7, 2023
Merged

[PEFT] Adapt example scripts to use PEFT#5388
sayakpaul merged 38 commits into
huggingface:mainfrom
younesbelkada:adapt-example-peft

Conversation

@younesbelkada

@younesbelkadayounesbelkada commented Oct 13, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes: #5341

Wandb logs: https://wandb.ai/younesbelkada/text2image-fine-tune/runs/tj5k8xtm?workspace=user-younesbelkada

I have fine-tuned it on Pokemon blip captions and the model trained after 1200 steps can be found here: https://huggingface.co/ybelkada/sd-1.5-pokemon-lora-peft

cc @sayakpaul@patrickvonplaten

TODO:

  • Update the SDXL LoRA script + add peft dependency on example tests workflow

younesbelkada
younesbelkada commented Oct 13, 2023
View reviewed changes
Comment threadexamples/text_to_image/train_text_to_image_lora.py
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Oct 13, 2023

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

sayakpaul
sayakpaul reviewed Oct 15, 2023
View reviewed changes
Comment threadexamples/text_to_image/train_text_to_image_lora.py
sayakpaul
sayakpaul reviewed Oct 15, 2023
View reviewed changes
Comment threadexamples/text_to_image/train_text_to_image_lora.py Outdated
@sayakpaul

sayakpaul commented Oct 15, 2023

Copy link
Copy Markdown
Member

Left some comments. Thanks for initiating this!

@patrickvonplaten

patrickvonplaten commented Oct 16, 2023

Copy link
Copy Markdown
Contributor

Before doing this change it would be good to have both a PEFT and a Transformers release to not force the user to install these packages from main

@younesbelkada

younesbelkada commented Oct 16, 2023

Copy link
Copy Markdown
ContributorAuthor

Before doing this change it would be good to have both a PEFT and a Transformers release to not force the user to install these packages from main

OK I will push the changes for SDXL and wait until the next release of PEFT before merging this PR

@younesbelkada

younesbelkada commented Oct 23, 2023

Copy link
Copy Markdown
ContributorAuthor

I can confirm the scripts work fine under peft + diffusers main! I'll mark the PR as ready for review once do a release on PEFT

@sayakpaulsayakpaul mentioned this pull request Nov 14, 2023
Merged
3 tasks
@younesbelkada
younesbelkada marked this pull request as ready for review November 14, 2023 15:44
@younesbelkada

younesbelkada commented Nov 14, 2023

Copy link
Copy Markdown
ContributorAuthor

This PR is ready for review! Let me know if you want me to work on saving adapter configs here, otherwise i can do it as a follow up PR. Please also see the attached wandb logs in the PR description above

sayakpaul
sayakpaul reviewed Nov 16, 2023
View reviewed changes
Comment threadexamples/dreambooth/requirements.txt Outdated
@sayakpaul

sayakpaul commented Nov 29, 2023

Copy link
Copy Markdown
Member

@patrickvonplaten this should be more or less ready to be merged, especially with huggingface/peft#1189.

@BenjaminBossan if would like to review the peft related changes introduced in this PR, that would be awesome.

Here's my Colab Notebook: https://colab.research.google.com/gist/sayakpaul/e112cc40288c5f0563efa4034dd50b72/scratchpad.ipynb.

sayakpaul
sayakpaul reviewed Nov 29, 2023
View reviewed changes
Comment threadexamples/dreambooth/train_dreambooth_lora.py
unet_lora_config = LoraConfig(
r=args.rank,
init_lora_weights="gaussian",
target_modules=["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"],

@sayakpaulsayakpaulNov 29, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@younesbelkada in your previous iterations, we were missing "to_out.0" in the target_module for some scripts. I have added that.

@sayakpaulsayakpaul mentioned this pull request Nov 29, 2023
Closed
@BenjaminBossan

BenjaminBossan commented Nov 29, 2023

Copy link
Copy Markdown
Member

From the PEFT-side of things, this looks pretty good, thanks Sayak.

Just a consideration: Right now, the requirements include a source install of PEFT. This works for the moment but could be dangerous in the future, in case we add something to PEFT main which breaks the examples (of course, that wouldn't be intentional, but these things can happen). So I wonder if it would be more cautious to fix the hash of the PEFT install and add a comment that we should change the dependency to a proper PEFT version once the next PEFT release is out (most likely, that would be >=0.7.0).

@sayakpaul

sayakpaul commented Nov 29, 2023

Copy link
Copy Markdown
Member

So I wonder if it would be more cautious to fix the hash of the PEFT install and add a comment that we should change the dependency to a proper PEFT version once the next PEFT release is out (most likely, that would be >=0.7.0).

@BenjaminBossan good point. Could you get me the required hash here? I will resolve ASAP.

@patrickvonplaten

patrickvonplaten commented Nov 29, 2023

Copy link
Copy Markdown
Contributor

I'd like to wait a bit until PEFT has done a release to make it easier for users (think it doesn't hurt us to wait a tad more here no?)

Does the training now match more or less (e.g. is the new init used?).

@BenjaminBossan

BenjaminBossan commented Nov 29, 2023

Copy link
Copy Markdown
Member

Could you get me the required hash here? I will resolve ASAP.

Latest hash is 8298f1a3668604ac9bc3f6e28b24e8eb554891a1, so adding @ + the hash to the requirement should do the trick. This is of course not required if we do:

I'd like to wait a bit until PEFT has done a release to make it easier for users

@sayakpaul

sayakpaul commented Nov 29, 2023

Copy link
Copy Markdown
Member

Does the training now match more or less (e.g. is the new init used?).

@patrickvonplaten yes it does. Have verified it. This PR uses the new init method from peft too. I don't mind waiting either.

This was referenced Dec 1, 2023
Closed
Closed
Closed
@younesbelkadayounesbelkada mentioned this pull request Dec 4, 2023
Closed
@younesbelkada

younesbelkada commented Dec 6, 2023

Copy link
Copy Markdown
ContributorAuthor

PEFT 0.7.0 has been released ! https://github.com/huggingface/peft/releases/tag/v0.7.0
I have updated the requirements file to reflect the correct PEFT version - requesting a final review cc @patrickvonplaten@sayakpaul

patrickvonplaten
patrickvonplaten reviewed Dec 6, 2023
View reviewed changes

@patrickvonplatenpatrickvonplaten left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me to add once @sayakpaul gives the green light

sayakpaul
sayakpaul approved these changes Dec 7, 2023
View reviewed changes

@sayakpaulsayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simply amazing here!

@sayakpaul
sayakpaul merged commit c271731 into huggingface:mainDec 7, 2023
@sayakpaulsayakpaul mentioned this pull request Dec 7, 2023
Closed
6 tasks
@younesbelkada
younesbelkada deleted the adapt-example-peft branch December 7, 2023 07:20
@sayakpaulsayakpaul mentioned this pull request Dec 10, 2023
Closed
donhardman pushed a commit to donhardman/diffusers that referenced this pull request Dec 18, 2023
@younesbelkada@sayakpaul@donhardman
[PEFT] Adapt example scripts to use PEFT (huggingface#5388)
43ff401
* adapt example scripts to use PEFT
* Update examples/text_to_image/train_text_to_image_lora.py
* fix
* add for SDXL
* oops
* make sure to install peft
* fix
* fix
* fix dreambooth and lora
* more fixes
* add peft to requirements.txt
* fix
* final fix
* add peft version in requirements
* remove comment
* change variable names
* add few lines in readme
* add to reqs
* style
* fix issues
* fix lora dreambooth xl tests
* init_lora_weights to gaussian and add out proj where missing
* ammend requirements.
* ammend requirements.txt
* add correct peft versions
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
@younesbelkadayounesbelkada mentioned this pull request Dec 18, 2023
Closed
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
@younesbelkada@sayakpaulJimmy
[PEFT] Adapt example scripts to use PEFT (huggingface#5388)
344616d
* adapt example scripts to use PEFT
* Update examples/text_to_image/train_text_to_image_lora.py
* fix
* add for SDXL
* oops
* make sure to install peft
* fix
* fix
* fix dreambooth and lora
* more fixes
* add peft to requirements.txt
* fix
* final fix
* add peft version in requirements
* remove comment
* change variable names
* add few lines in readme
* add to reqs
* style
* fix issues
* fix lora dreambooth xl tests
* init_lora_weights to gaussian and add out proj where missing
* ammend requirements.
* ammend requirements.txt
* add correct peft versions
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sayakpaulsayakpaulsayakpaul approved these changes
+1 more reviewer
@patrickvonplatenpatrickvonplatenpatrickvonplaten left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[PEFT] Update the example scripts to reflect PEFT API

5 participants

@younesbelkada@HuggingFaceDocBuilderDev@sayakpaul@patrickvonplaten@BenjaminBossan