support sd3.5 for controlnet example by DavyMorgan · Pull Request #9860 · huggingface/diffusers

DavyMorgan · 2024-11-04T15:14:30Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

DavyMorgan · 2024-11-04T15:33:19Z

@sayakpaul Hi, in this PR, I add sd3.5 to the controlnet example, with only a few lines added!

yiyixuxu

thanks!

HuggingFaceDocBuilderDev · 2024-11-04T18:20:08Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu · 2024-11-04T18:52:51Z

@DavyMorgan can you fix the test?

DavyMorgan · 2024-11-04T19:13:04Z

@DavyMorgan can you fix the test?

Yeah, I have updated the test.
@yiyixuxu @sayakpaul Could you please run the test workflows again?

yiyixuxu · 2024-11-08T01:38:07Z

@DavyMorgan
would you be able to update the branch to sync with the main (and resolve the conflicts)?
sorry about that!

DavyMorgan · 2024-11-15T13:29:05Z

@DavyMorgan would you be able to update the branch to sync with the main (and resolve the conflicts)? sorry about that!

Sure, I can fix that.

DavyMorgan · 2024-11-17T02:21:02Z

@yiyixuxu @sayakpaul Could you please run the test workflows again?

DavyMorgan · 2024-11-17T03:01:33Z

@sayakpaul I updated the codes. This time it should be ok. :)

sayakpaul

Thanks for the changes!

Could you provide some example results as well?

Additionally, I hosted the tiny pipe to our testing org:
https://huggingface.co/hf-internal-testing/tiny-sd35-pipe

Could you please change the path accordingly in test_controlnet.py?

I will let @yiyixuxu review the other changes introduced.

sayakpaul · 2024-11-17T03:06:00Z

 else:
 logger.info("Initializing controlnet weights from transformer")
- controlnet = SD3ControlNetModel.from_transformer(transformer)
+ controlnet = SD3ControlNetModel.from_transformer(transformer, num_extra_conditioning_channels=0)


Should this be hard-coded like this? Can we use a CLI arg?

Thanks, I have added a CLI arg.

DavyMorgan · 2024-11-17T03:27:28Z

https://huggingface.co/hf-internal-testing/tiny-sd35-pipe

@sayakpaul I have updated the codes to use hf-internal-testing/tiny-sd35-pipe. Meanwhile, the example results can be found at the end of examples/controlnet/README_sd3.md.

yiyixuxu · 2024-11-20T06:09:51Z

 # controlnet residual
 if block_controlnet_hidden_states is not None and block.context_pre_only is False:
- interval_control = len(self.transformer_blocks) // len(block_controlnet_hidden_states)
+ interval_control = int(math.ceil(len(self.transformer_blocks) / len(block_controlnet_hidden_states)))


sorry I missed this change - any reason we make this change here?

@yiyixuxu Yes, this change is very much related to #9758 and also inspired by the flux-controlnet implementation, which avoids crash when transitioning from sd3 to sd3.5.

yiyixuxu · 2024-11-20T10:09:06Z

related ( a little bit overlap with) #9758

DavyMorgan · 2024-11-27T17:00:37Z

@yiyixuxu @sayakpaul Can you run the tests again to check whether it works well with the current HEAD?

sayakpaul · 2024-11-28T11:00:28Z

@DavyMorgan sorry for the late reply. Could you consider the changes introduced in #9758 and repurpose this PR accordingly?

yiyixuxu · 2024-12-03T03:13:02Z

 pooled_projection_dim: int = 2048,
 out_channels: int = 16,
 pos_embed_max_size: int = 96,
+ dual_attention_layers: Tuple[


we already have this change
diffusers/src/diffusers/models/controlnets/controlnet_sd3.py
Line 60 in 963ffca
dual_attention_layers: Tuple[int, ...] = (),
can you rebase?

@yiyixuxu done.

DavyMorgan · 2024-12-05T16:09:57Z

@yiyixuxu @sayakpaul Hi, I have fixed the codes to pass the quality test. Please run the tests again!

DavyMorgan · 2024-12-06T02:33:00Z

Sorry for missing the typo😂Now fixed. @yiyixuxu @sayakpaul

* support sd3.5 in controlnet --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

DavyMorgan added 8 commits November 4, 2024 10:10

support sd3.5 in controlnet
016367d

fix train controlnet sd3 from transformer
41664d6

add test script for training controlnet sd3.5
f35a31e

fix controlnet
b9b7028

add sd3.5 to readme
6ac4251

add preprocessing batchsize
fb1ae0c

update readme
dbd60e3

pass make quality
9765601

sayakpaul requested a review from yiyixuxu November 4, 2024 17:06

yiyixuxu approved these changes Nov 4, 2024
View reviewed changes

update test
4f8b3b0

Merge branch 'main' into sd3.5-controlnet
88eb200

fix sd3.5
c0f1a87

sayakpaul reviewed Nov 17, 2024
View reviewed changes

add extra conditioning channels to cli, use hf-internal-testing host
df9712b

yiyixuxu reviewed Nov 20, 2024
View reviewed changes

yiyixuxu added the close-to-merge label Nov 20, 2024

Merge branch 'main' into sd3.5-controlnet
5c2709a

Merge branch 'main' into sd3.5-controlnet
3220b71

yiyixuxu reviewed Dec 3, 2024
View reviewed changes

DavyMorgan added 2 commits December 2, 2024 22:18

rebase
13c8d13

rebase
f143065

xduzhangjiayu mentioned this pull request Dec 5, 2024
Training script for a Controlnet based on SD3 does not work #10055
Open

quality
3a06a97

DavyMorgan added 2 commits December 5, 2024 12:58

Merge branch 'huggingface:main' into sd3.5-controlnet
b135b97

fix typo
0ac2192

yiyixuxu merged commit 6131a93 into huggingface:mainDec 6, 2024

yiyixuxu removed the close-to-merge label Dec 6, 2024

sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
support sd3.5 for controlnet example (#9860)
3271d38
* support sd3.5 in controlnet --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

Uh oh!

Conversation

DavyMorgan commented Nov 4, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

DavyMorgan commented Nov 4, 2024

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Nov 4, 2024

Uh oh!

yiyixuxu commented Nov 4, 2024

Uh oh!

DavyMorgan commented Nov 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu commented Nov 8, 2024

Uh oh!

DavyMorgan commented Nov 15, 2024

Uh oh!

DavyMorgan commented Nov 17, 2024

Uh oh!

DavyMorgan commented Nov 17, 2024

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaulNov 17, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorganNov 17, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorgan commented Nov 17, 2024

Uh oh!

yiyixuxuNov 20, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorganNov 20, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Nov 20, 2024

Uh oh!

DavyMorgan commented Nov 27, 2024

Uh oh!

sayakpaul commented Nov 28, 2024

Uh oh!

yiyixuxuDec 3, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorganDec 3, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorganDec 3, 2024

Choose a reason for hiding this comment

Uh oh!

DavyMorgan commented Dec 5, 2024

Uh oh!

DavyMorgan commented Dec 6, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DavyMorgan commented Nov 4, 2024•
edited
Loading

DavyMorgan commented Dec 6, 2024•
edited
Loading