AttributeError: _hf_hook caused by delattr in hooks.remove_hook_from_module()

[eppaneamd]

System Info

- `Accelerate` version: 1.4.0.dev0
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.39
- `accelerate` bash location: /workspaces/.venv_py311/bin/accelerate
- Python version: 3.11.11
- Numpy version: 1.26.3
- PyTorch version (GPU?): 2.5.1+rocm6.2 (True)
- System RAM: 1007.70 GB
- GPU type: AMD Instinct MI250X/MI250
- `Accelerate` default config:
 Not found

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

When using the HunyuanVideo model via diffusers framework with the following script:

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel

MODEL_ID = "hunyuanvideo-community/HunyuanVideo"
PROMPT = "A cat walks on the grass, realistic"

transformer = HunyuanVideoTransformer3DModel.from_pretrained(
 MODEL_ID, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(
 MODEL_ID, transformer=transformer, torch_dtype=torch.float16
)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()

pipe.transformer = torch.compile(pipe.transformer, mode="default")

# Warmup
_ = pipe(
 prompt=PROMPT,
 height=320,
 width=512,
 num_frames=61,
 num_inference_steps=1,
 max_sequence_length=256,
 guidance_scale=0.0,
 generator=torch.Generator(device="cuda").manual_seed(42),
)

# Inference
output = pipe(
 prompt=PROMPT,
 height=320,
 width=512,
 num_frames=61,
 num_inference_steps=30,
 max_sequence_length=256,
 guidance_scale=0.0,
 output_type="pil",
 generator=torch.Generator(device="cuda").manual_seed(42),
)

Following error arises:

Traceback (most recent call last):
 File "/workspaces/huvideo_diffusers_repro.py", line 21, in <module>
 _ = pipe(
 ^^^^^
 File "/workspaces/.venv_py311/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
 return func(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^
 File "/workspaces/diffusers/src/diffusers/pipelines/hunyuan_video/pipeline_hunyuan_video.py", line 689, in __call__
 self.maybe_free_model_hooks()
 File "/workspaces/diffusers/src/diffusers/pipelines/pipeline_utils.py", line 1119, in maybe_free_model_hooks
 self.enable_model_cpu_offload(device=getattr(self, "_offload_device", "cuda"))
 File "/workspaces/diffusers/src/diffusers/pipelines/pipeline_utils.py", line 1050, in enable_model_cpu_offload
 self.remove_all_hooks()
 File "/workspaces/diffusers/src/diffusers/pipelines/pipeline_utils.py", line 1017, in remove_all_hooks
 accelerate.hooks.remove_hook_from_module(model, recurse=True)
 File "/workspaces/accelerate/src/accelerate/hooks.py", line 203, in remove_hook_from_module
 delattr(module, "_hf_hook")
 File "/workspaces/.venv_py311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2043, in __delattr__
 super().__delattr__(name)
AttributeError: 'OptimizedModule' object has no attribute '_hf_hook'

When wrapping these two lines with try ... except, the run succeeds.

https://github.com/huggingface/accelerate/blob/main/src/accelerate/hooks.py#L203
https://github.com/huggingface/accelerate/blob/main/src/accelerate/hooks.py#L212

If we still log the exceptions, they produce:

Error: AttributeError – 'OptimizedModule' object has no attribute '_hf_hook'
Error: AttributeError – 'OptimizedModule' object has no attribute '_old_forward'

This indicates that there seems to be a logical error when using the following coding pattern:

if hasattr(object, attr):
 delattr(object, attr)

Suggesting that even though hasattr returns True, there is no guarantee delattr will work as intended.

Expected behavior

Successful run without errors when using torch.compile and cpu model offloading.

[eppaneamd]

@muellerzr @a-r-r-o-w

[eppaneamd]

I apparently had an older version of accelerate. I recloned and reinstalled the repo again from source.

I have updated the issue accordingly. Added also a repro code in plain diffusers for simplicity.

[eppaneamd]

Another repro with Flux:

import torch
from diffusers import FluxPipeline

MODEL_ID = "black-forest-labs/FLUX.1-dev"
PROMPT = "A small cat"

pipe = FluxPipeline.from_pretrained(pretrained_model_name_or_path=MODEL_ID, torch_dtype=torch.bfloat16)

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()

pipe.transformer = torch.compile(pipe.transformer, mode="default")

# Warmup
_ = pipe(
 prompt=PROMPT,
 height=1024,
 width=1024,
 num_inference_steps=1,
 max_sequence_length=256,
 guidance_scale=0.0,
 generator=torch.Generator(device="cuda").manual_seed(42),
)

# Inference
output = pipe(
 prompt=PROMPT,
 height=1024,
 width=1024,
 num_inference_steps=25,
 max_sequence_length=256,
 guidance_scale=0.0,
 output_type="pil",
 generator=torch.Generator(device="cuda").manual_seed(42),
)

[SunMarc]

Thanks for the repo ! I'll have a look soon !

[Yukun-Huang]

@eppaneamd Same issue! Is there any temporary solution for this?

[eppaneamd]

@Yukun-Huang you can do that try ... except band-aid fix described above. Something like this in practice:

@@ -200,7 +200,10 @@ def remove_hook_from_module(module: nn.Module, recurse=False):
 
 if hasattr(module, "_hf_hook"):
 module._hf_hook.detach_hook(module)
- delattr(module, "_hf_hook")
+ try:
+ delattr(module, "_hf_hook")
+ except AttributeError as e:
+ print("Error: ", type(e).__name__, "–", e)
 
 if hasattr(module, "_old_forward"):
 # Overriding a GraphModuleImpl forward freezes the forward call and later modifications on the graph will fail.
@@ -209,7 +212,10 @@ def remove_hook_from_module(module: nn.Module, recurse=False):
 module.__class__.forward = module._old_forward
 else:
 module.forward = module._old_forward
- delattr(module, "_old_forward")
+ try:
+ delattr(module, "_old_forward")
+ except AttributeError as e:
+ print("Error: ", type(e).__name__, "–", e)
 
 # Remove accelerate added warning hooks from dispatch_model
 for attr in _accelerate_added_attributes:

[Yukun-Huang]

@eppaneamd Thanks for your sharing! And looking forward to the official fix.

[SunMarc]

Transfered here as it is an issue with diffusers. The issue is that we are trying to remove the hook from the compiled model which trigger this error. The fix is quite easy (just unwrap the model) but I wanted to know Is it common to compile the model after we used enable_model_cpu_offload cc @sayakpaul ?

EDIT: there are more issues later that i don't know how to fix

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.