Make IPAdapter compatible to torch.compile

[rootonchair]

What does this PR do?

Fixes#7985

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[yiyixuxu]

should we also add a test?

[yiyixuxu]

ok but do we know why it has to be done this way? any documentation etc?
e.g. would this work?

forlayerinself.layers:
[ln0, ln1, attn, ff] =layers

[rootonchair]

If you try to unpack a ModuleList like above or just like in the original implementation. Following error occurs:

torch._dynamo.exc.Unsupported: UNPACK_SEQUENCE NNModuleVariable()

So I assume that any unpacking with ModuleList is not supported in torch.compile and only for loop through it is supported. Hence, I for loop the ModuleList to not breaking the graph and upack using python list

[rootonchair]

I also try moving ln0, ln1, attn, ff into a class, but it will affect the weight loading

[fabiorigano]

I also try moving ln0, ln1, attn, ff into a class, but it will affect the weight loading

hi @rootonchair, you can use the IPAdapterPlusImageProjectionBlock class, but you have to update the loading function too.
I think the final result would be really good, as the code consistency in the library would improve

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

cc @fabiorigano too

[fabiorigano]

hi @yiyixuxu thanks for adding me here
IPAdapterPlusImageProjection was added some time ago, I am not sure why it has been implemented in this way

I think it would be nice to have a forward function that uses the

diffusers/src/diffusers/models/embeddings.py

Line 892 in d6ca120

classIPAdapterPlusImageProjectionBlock(nn.Module):

that we introduced with the Face ID PR #7186

[yiyixuxu]

thanks @fabiorigano!
cc @rootonchair, does face id currently work with torch.compile?

[rootonchair]

@yiyixuxu yes it does. As @fabiorigano suggested, we could utilize IPAdapterPlusImageProjectionBlock however weight loading also need to be updated

[sayakpaul]

This works for me. Thank you, @rootonchair!

Perhaps the use of IPAdapterPlusImageProjectionBlock could live in a different PR? @yiyixuxu WDYT?

[fabiorigano]

I can open a new PR to use IPAdapterPlusImageProjectionBlock, if @rootonchair agrees

[sayakpaul]

That would be great, thank you!

[rootonchair]

Sure @fabiorigano, thank you

[sayakpaul]

@rootonchair does #7994 work for you?

[rootonchair]

@sayakpaul yes it does. Fantastic works

[sayakpaul]

Fantastic. Would you mind closing the PR then? I am sorry about the inconvenience here.

[rootonchair]

Ah, no problem. I assume closing #7985 too?