Allow configuring `shift=` for SD3 dynamically

[isidentical]

Is your feature request related to a problem? Please describe.
Allow passing shift= per inference call (like timesteps) on the pipeline, for flow matching scheduler, or allow set_shift() etc. on the scheduler. This seems to be the key to getting good results with SD3 https://x.com/bfitzgerald242/status/1801018438120341911

[asomoza]

Hi, you can do it like this:

from diffusers import FlowMatchEulerDiscreteScheduler

pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_config(pipe.scheduler.config, shift=3.0)

[isidentical]

yep! but the same format is applicable for timesteps and was wondering if we can get around without re-instating the scheduler again and again?

[asomoza]

Not for the moment, but I can see the potential in adding it as an argument if people change it a lot for each inference.

In my experience it didn't fix the anatomy problems and sometimes it made the quality worse but I tested it with the T5, still need to test it without it and do some more generations.

Can you share some examples where changing the shift helped with the generation? That would help a lot.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[hlky]

Useful to have, changing shift is popular with HunyuanVideo and SD3/Flux.

[bghira]

does it really help though? examples requested never were shared.

[hlky]

Documented benefit for HunyuanVideo.

diffusers/docs/source/en/api/pipelines/hunyuan_video.md

Line 32 in 1524781

- For smaller resolution images, try lower values of `shift` (between `2.0` to `5.0`) in the [Scheduler](https://huggingface.co/docs/diffusers/main/en/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler.shift). For larger resolution images, try higher values (between `7.0` and `12.0`). The default value is `7.0` for HunyuanVideo.

I'll run some tests for SD3/Flux to confirm.

[bghira]

but SD3 already has resolution-dependent shift using the mu calculations, right?

[hlky]

Actually this won't do anything for Flux because of dynamic shifting. We recently added support for dynamic shifting in SD3, it's not used by default though. Either way, it's a simple change that has at least some benefit to HunyuanVideo and it won't do any harm to add a function to change shift without creating the scheduler each time for those that want it.