fix: missing AutoencoderKL lora adapter by beniz · Pull Request #9807 · huggingface/diffusers

beniz · 2024-10-30T07:43:20Z

What does this PR do?

This PR fixes the missing lora adapter with VAE (AutoencoderKL class).
Discussion is here: #9771
Related reports:
GaParmar/img2img-turbo#64
radames/Real-Time-Latent-Consistency-Model#38

Who can review?

cc @sayakpaul

sayakpaul

Thank you! Just a single comment.

HuggingFaceDocBuilderDev · 2024-11-06T01:50:01Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu · 2024-11-06T02:19:12Z

can you run make style?

yiyixuxu · 2024-11-08T07:33:33Z

the failing tests are related, can we look into them?

I think we may need to add a @require_peft_backend similar to

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 1081 in 5588725

deftest_lora(self):

sayakpaul

Sorry I had actually forgot to submit my reviews.

sayakpaul · 2024-11-02T14:54:13Z


 from ..test_modeling_common import ModelTesterMixin, UNetTesterMixin
-
+from peft import LoraConfig


This needs to be guarded like this:
diffusers/tests/models/unets/test_models_unet_2d_condition.py
Line 56 in 13e8fde
ifis_peft_available():

sayakpaul · 2024-11-02T14:54:37Z


 self.assertTrue(torch_all_close(output_slice, expected_output_slice, rtol=1e-2))

+ def test_lora_adapter(self):


This needs to be decorated with:
diffusers/tests/models/unets/test_models_unet_2d_condition.py
Line 1080 in 13e8fde
@require_peft_backend

@beniz seems like this was not resolved?

@sayakpaul ah apologies, may have forgot to push to repo. Done. Thanks for your vigilance.

beniz · 2024-11-11T14:32:15Z

the failing tests are related, can we look into them?
I think we may need to add a @require_peft_backend similar to
diffusers/tests/models/unets/test_models_unet_2d_condition.py
Line 1081 in 5588725
deftest_lora(self):

All done @yiyixuxu I believe, let me know if anything else remains.

sayakpaul · 2024-11-29T02:14:10Z

@beniz thanks! Possible to fix the quality issues by running make style && make quality?

beniz · 2024-11-29T10:59:19Z

@beniz thanks! Possible to fix the quality issues by running make style && make quality?

I've fix a missing dependency. FYI running make style && qualilty fails early on other unrelated files, so I've been looking at the underlying ruff calls to apply them to the relevant files.

sayakpaul · 2024-11-29T11:28:21Z

@beniz I pushed the quality fixes directly to your branch, which I hope is okay. If not, please let me know, I will revert immediately

beniz · 2024-11-29T11:39:54Z

@beniz I pushed the quality fixes directly to your branch, which I hope is okay. If not, please let me know, I will revert immediately

Much appreciated, thank you.

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

…h bnb components (#9840) * allow device placement when using bnb quantization. * warning. * tests * fixes * docs. * require accelerate version. * remove print. * revert to() * tests * fixes * fix: missing AutoencoderKL lora adapter (#9807) * fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fixes * fix condition test * updates * updates * remove is_offloaded. * fixes * better * empty --------- Co-authored-by: Emmanuel Benazera <emmanuel.benazera@jolibrain.com>

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

…h bnb components (#9840) * allow device placement when using bnb quantization. * warning. * tests * fixes * docs. * require accelerate version. * remove print. * revert to() * tests * fixes * fix: missing AutoencoderKL lora adapter (#9807) * fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fixes * fix condition test * updates * updates * remove is_offloaded. * fixes * better * empty --------- Co-authored-by: Emmanuel Benazera <emmanuel.benazera@jolibrain.com>

sayakpaul reviewed Oct 30, 2024
View reviewed changes

Comment threadsrc/diffusers/models/autoencoders/autoencoder_kl.py

beniz mentioned this pull request Oct 30, 2024
chore: diffusers update jolibrain/joliGEN#719
Merged

beniz force-pushed the fix-vae-lora branch from c4f243d to f4c8d6cCompare October 30, 2024 08:15

sayakpaul reviewed Oct 30, 2024
View reviewed changes

Comment threadtests/models/autoencoders/test_models_vae.py Outdated

beniz force-pushed the fix-vae-lora branch 4 times, most recently from 09cd44d to 9fb4880Compare November 2, 2024 11:42

beniz force-pushed the fix-vae-lora branch from 9fb4880 to ca918e4Compare November 6, 2024 13:31

sayakpaul reviewed Nov 8, 2024
View reviewed changes

beniz force-pushed the fix-vae-lora branch from 591af15 to ca918e4Compare November 8, 2024 16:08

beniz force-pushed the fix-vae-lora branch from c7b60f5 to 76b2277Compare November 28, 2024 21:18

fix: missing AutoencoderKL lora adapter
7025526

beniz force-pushed the fix-vae-lora branch from da99538 to 7025526Compare November 29, 2024 08:26

sayakpaul added 2 commits November 29, 2024 16:47

Merge branch 'main' into fix-vae-lora
1a5e634

fix
ecda350

sayakpaul approved these changes Nov 29, 2024
View reviewed changes

sayakpaul requested a review from yiyixuxu November 29, 2024 11:48

sayakpaul added the close-to-merge label Nov 29, 2024

yiyixuxu merged commit 963ffca into huggingface:mainDec 3, 2024

sayakpaul added a commit that referenced this pull request Dec 3, 2024
fix: missing AutoencoderKL lora adapter (#9807)
1fe8a79
* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

lawrence-cj pushed a commit to lawrence-cj/diffusers that referenced this pull request Dec 4, 2024
fix: missing AutoencoderKL lora adapter (huggingface#9807)
f379ac2
* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

sayakpaul added a commit that referenced this pull request Dec 23, 2024
fix: missing AutoencoderKL lora adapter (#9807)
3dd8aa4
* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>


		from ..test_modeling_common import ModelTesterMixin, UNetTesterMixin

		from peft import LoraConfig


		self.assertTrue(torch_all_close(output_slice, expected_output_slice, rtol=1e-2))

		def test_lora_adapter(self):

Uh oh!

Conversation

beniz commented Oct 30, 2024

What does this PR do?

Who can review?

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 6, 2024

Uh oh!

yiyixuxu commented Nov 6, 2024

Uh oh!

yiyixuxu commented Nov 8, 2024

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaulNov 2, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaulNov 2, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaulNov 28, 2024

Choose a reason for hiding this comment

Uh oh!

benizNov 28, 2024

Choose a reason for hiding this comment

Uh oh!

beniz commented Nov 11, 2024

Uh oh!

sayakpaul commented Nov 29, 2024

Uh oh!

beniz commented Nov 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Nov 29, 2024

Uh oh!

beniz commented Nov 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

beniz commented Nov 29, 2024•
edited
Loading