Fix: leaf_level offloading breaks after delete_adapters

[adi776borate]

What does this PR do?

Fixes#12396

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Discussed in the issue Using leaf_level together with delete_adapters can get some errors. #12396.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@yiyixuxu
@sayakpaul
Anyone in the community is free to review the PR once the tests have passed.

[yiyixuxu]

thanks for the PR! @adi776borate
can you check if we have same problem with other offloading methods? e.g. enable_model_cpu_offload and enable_sequential_cpu_offload

[sayakpaul]

@adi776borate thanks for your PR! Could you also add a test for this in https://github.com/huggingface/diffusers/blob/main/tests/lora/utils.py?

@yiyixuxu we have a test for enable_model_cpu_offload():

diffusers/tests/lora/utils.py

Line 2341 in 7a001c3

deftest_lora_loading_model_cpu_offload(self):

I ran it with pytest tests/lora/ -k "model_cpu_offload" and got:

====================== 16 passed, 1 skipped, 860 deselected, 2 warnings in 60.04s (0:01:00) ======================

So, @adi776borate maybe we could start by adding a similar test case as

diffusers/tests/lora/utils.py

Line 2341 in 7a001c3

deftest_lora_loading_model_cpu_offload(self):

but for enable_sequential_cpu_offload() (can happen in a separate PR).

[adi776borate]

@sayakpaul I've added a regression test test_lora_group_offloading_delete_adapters to tests/lora/utils.py as requested. It reproduces the crash reported and ensures that the next inference pass doesn't fail.

@yiyixuxu I also verified locally that enable_sequential_cpu_offload and enable_model_cpu_offload do not have this issue. They work correctly with delete_adapters.

Ready for the review.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

We can make this test tighter by also adding some assertions:

diff --git a/tests/lora/utils.py b/tests/lora/utils.py
index efe68dd3c..7e19aac29 100644
--- a/tests/lora/utils.py+++ b/tests/lora/utils.py@@ -2403,7 +2403,9 @@ class PeftLoraLoaderMixinTests:
)
pipe.load_lora_weights(tmpdirname, adapter_name="default")
- pipe(**inputs, generator=torch.manual_seed(0))+ out_lora = pipe(**inputs, generator=torch.manual_seed(0))[0]+
# Delete the adapter
pipe.delete_adapters("default")
- pipe(**inputs, generator=torch.manual_seed(0))
\ No newline at end of file
+ out_no_lora = pipe(**inputs, generator=torch.manual_seed(0))[0]+ self.assertFalse(np.allclose(out_lora, out_no_lora, atol=1e-3, rtol=1e-3))
\ No newline at end of file

[adi776borate]

Thanks for the suggestion! I have made the changes.

[sayakpaul]

Thanks!

[adi776borate]

@yiyixuxu I'm getting familiar with testing and resolving CI errors.
Please review the latest commit.
Thank You.

[sayakpaul]

I have ran the tests and they are passing as well. Thanks for your work!

[sayakpaul]

@yiyixuxu a gentle ping.

@songh11 could you check if this fixes #12396?

[songh11]

@yiyixuxu a gentle ping. 一个轻柔的提示。

@songh11 could you check if this fixes #12396?你能检查这个是否修复了 #12396 吗？

Sorry for being busy recently and not seeing this earlier—this solved my problem perfectly. Thank you! @adi776borate

[sayakpaul]

Failing tests are unrelated.

[sayakpaul]

Thanks for your contributions, @adi776borate!

[adi776borate]

Thanks for your contributions, @adi776borate!

Thank you @sayakpaul and @yiyixuxu! I really appreciate the guidance. It's been a great experience, and I'm excited to have my first PR merged into 🧨.