Is your feature request related to a problem? Please describe.
Stable Diffusion UNet inference is FLOP-heavy, making generation costly on constrained hardware. Clockwork Diffusion (arXiv:2312.08128) is a training-free, inference-time acceleration for diffusion UNets based on the observation that high-resolution UNet feature maps are perturbation-sensitive, while low-resolution maps are stable across adjacent denoising steps. By periodically reusing (caching) low-res feature maps from a preceding step to approximate them at one or more subsequent steps — skipping their recomputation — Stable Diffusion v1.5 with 8 DPM++ steps saves ~32% FLOPs with negligible FID and CLIP change, for both text-to-image generation and image editing and across multiple baselines. There are no new weights, no trainer, and no new data format.
Describe the solution you'd like.
Implement clockwork feature-map caching behind an opt-in flag on a Stable Diffusion v1.5 pipeline. Mechanically this requires (a) splitting UNet blocks into high-res vs low-res, (b) caching the low-res outputs, and (c) replaying them across scheduler timesteps — a purely inference-side modification, broadly compatible with diffusers.
Suggested first scope: a single baseline (SD1.5) + single scheduler (DPM++ 8-step) + one cache cadence, then benchmark against the uncached baseline on a fixed prompt set (e.g. a COCO-2017 caption or PartiPrompts subset): report wall-clock and FLOPs savings plus FID and CLIP-score delta, targeting the paper's ~32% FLOPs reduction at negligible FID/CLIP change. Proposed acceptance criterion: reproduce that savings before merging.
Describe alternatives you've considered.
- Host module: extend
UNet2DConditionModel.forward with an opt-in clockwork path, or add a dedicated wrapper/custom pipeline (e.g. a ClockworkDiffusionPipeline) that keeps the change isolated. Unlike toggleable inference tweaks already in diffusers (e.g. enable_freeu), Clockwork must carry cross-step state plus UNet-forward surgery, so neither option is a drop-in — which does the team prefer?
- Scope: defer the paper's full multi-baseline and image-editing sweeps to follow-ups rather than tackling them in the first PR.
- There is no released artifact (benchmark or checkpoint) to route through an existing eval path; the caching logic has to be re-derived from the method itself.
Additional context.
Why this is an Issue rather than a PR. The recommendation carried no candidate-selection mapping, no named call site, and no scoped slice to evaluate — the visible repo layout is the conversion-only scripts/ directory, none of which would host a feature-caching inference path. Clockwork is not a drop-in or a consumable released artifact: it is an inference-internals change with no identified host module, so the team should agree on a host and scope before a PR is opened.
Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.
Drafted by Outrider — paper: arXiv:2312.08128.
Discovery context
Recommended paper: Clockwork Diffusion: Efficient Generation With Model-Step Distillation
Confidence: moderate (Remyx relevance 0.65)
Research interest: (pin-method)
TL;DR: Clockwork caches low-res UNet feature maps across denoising steps to cut ~32% FLOPs (SD1.5, 8 DPM++ steps) at negligible FID/CLIP loss. Inference-only but needs UNet-forward + cross-step-state surgery; no call site named — pick a host before PR.
Why the orchestrator opened an Issue instead of a PR. Pre-flight routed to Issue before implementation: the spec ships with no selection rationale, no named call site, and no suggested experiment, so there is no scoped slice to evaluate. That is an inference-internals change with no identified host module (the visible layout is the conversion-only scripts/ directory), so the team should agree on a host and scope before a PR.
Opened by the Remyx Recommendation orchestrator. Pre-flight routed this paper to Issue before the coding agent ran — see the reasoning above for what would need to change to scaffold it as a PR.
Is your feature request related to a problem? Please describe.
Stable Diffusion UNet inference is FLOP-heavy, making generation costly on constrained hardware. Clockwork Diffusion (arXiv:2312.08128) is a training-free, inference-time acceleration for diffusion UNets based on the observation that high-resolution UNet feature maps are perturbation-sensitive, while low-resolution maps are stable across adjacent denoising steps. By periodically reusing (caching) low-res feature maps from a preceding step to approximate them at one or more subsequent steps — skipping their recomputation — Stable Diffusion v1.5 with 8 DPM++ steps saves ~32% FLOPs with negligible FID and CLIP change, for both text-to-image generation and image editing and across multiple baselines. There are no new weights, no trainer, and no new data format.
Describe the solution you'd like.
Implement clockwork feature-map caching behind an opt-in flag on a Stable Diffusion v1.5 pipeline. Mechanically this requires (a) splitting UNet blocks into high-res vs low-res, (b) caching the low-res outputs, and (c) replaying them across scheduler timesteps — a purely inference-side modification, broadly compatible with diffusers.
Suggested first scope: a single baseline (SD1.5) + single scheduler (DPM++ 8-step) + one cache cadence, then benchmark against the uncached baseline on a fixed prompt set (e.g. a COCO-2017 caption or PartiPrompts subset): report wall-clock and FLOPs savings plus FID and CLIP-score delta, targeting the paper's ~32% FLOPs reduction at negligible FID/CLIP change. Proposed acceptance criterion: reproduce that savings before merging.
Describe alternatives you've considered.
UNet2DConditionModel.forwardwith an opt-in clockwork path, or add a dedicated wrapper/custom pipeline (e.g. aClockworkDiffusionPipeline) that keeps the change isolated. Unlike toggleable inference tweaks already in diffusers (e.g.enable_freeu), Clockwork must carry cross-step state plus UNet-forward surgery, so neither option is a drop-in — which does the team prefer?Additional context.
Why this is an Issue rather than a PR. The recommendation carried no candidate-selection mapping, no named call site, and no scoped slice to evaluate — the visible repo layout is the conversion-only
scripts/directory, none of which would host a feature-caching inference path. Clockwork is not a drop-in or a consumable released artifact: it is an inference-internals change with no identified host module, so the team should agree on a host and scope before a PR is opened.Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.
Drafted by Outrider — paper: arXiv:2312.08128.
Discovery context
Recommended paper: Clockwork Diffusion: Efficient Generation With Model-Step Distillation
Confidence: moderate (Remyx relevance 0.65)
Research interest: (pin-method)
TL;DR: Clockwork caches low-res UNet feature maps across denoising steps to cut ~32% FLOPs (SD1.5, 8 DPM++ steps) at negligible FID/CLIP loss. Inference-only but needs UNet-forward + cross-step-state surgery; no call site named — pick a host before PR.
Why the orchestrator opened an Issue instead of a PR. Pre-flight routed to Issue before implementation: the spec ships with no selection rationale, no named call site, and no suggested experiment, so there is no scoped slice to evaluate. That is an inference-internals change with no identified host module (the visible layout is the conversion-only
scripts/directory), so the team should agree on a host and scope before a PR.Opened by the Remyx Recommendation orchestrator. Pre-flight routed this paper to Issue before the coding agent ran — see the reasoning above for what would need to change to scaffold it as a PR.