Marigold Update: v1-1 models, Intrinsic Image Decomposition pipeline, documentation by toshas · Pull Request #10884 · huggingface/diffusers

toshas · 2025-02-23T23:39:18Z

What does this PR do?

This PR updates and refactors the Marigold pipelines to support the new v1-1 models, enabling fast inference (1-4 steps) with DDIM. It also introduces a new Intrinsic Image Decomposition (IID) pipeline.

Fixes#10883

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@sayakpaul @yiyixuxu

fix missing prediction_type in register_to_config in the normals pipeline

update comments about num_inference_steps and ensemble_size minor fixes in comments of marigold normals and depth pipelines

HuggingFaceDocBuilderDev · 2025-02-24T04:18:20Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

toshas · 2025-02-24T10:11:16Z

Looks like most failures come from the kolors pipeline - do I need to do anything about it?

a-r-r-o-w · 2025-02-24T10:28:23Z

@toshas Thanks for the PR & congrats on the release! The failing tests are unrelated and have been failing across all PRs. I'll try to review the full changes soon!

stevhliu

Very nice and thorough docs! 👏

yiyixuxu

thanks for the PR! looks good to me!

a-r-r-o-w

thanks! could you run make style and make fix-copies for the quality tests to pass

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

…e base

compute n_targets on the go add a consistency check for vae_scale_factor

move quantitative evaluation and predictive uncertainty sections of marigold into the end

toshas · 2025-02-25T02:11:57Z

I finished the improvements; please let me know if there is anything else. Thanks!

yiyixuxu · 2025-02-26T00:13:21Z

thanks a lot! @toshas

nitinmukesh · 2025-02-26T09:24:22Z

Hello @toshas

When are you planning to release the new models
https://huggingface.co/prs-eth/marigold-depth-v1-1

toshas · 2025-05-15T17:35:04Z

The models have been released! @nitinmukesh

toshas added 9 commits February 17, 2025 22:07

minor documentation fixes of the depth and normals pipelines
a490417

update license headers
d002d5a

update model checkpoints in examples
40e2040
fix missing prediction_type in register_to_config in the normals pipeline

add initial marigold intrinsics pipeline
2f0bbbd
update comments about num_inference_steps and ensemble_size minor fixes in comments of marigold normals and depth pipelines

update uncertainty visualization to work with intrinsics
d20642c

integrate iid
e007863

add marigold intrinsics tests
6b5267f

update documentation
caa4a62

Merge branch 'main' into pipeline_marigold_intrinsics
5a4196e

toshas mentioned this pull request Feb 23, 2025
Marigold Update: v1-1 models, Intrinsic Image Decomposition pipeline, documentation #10883
Closed
2 tasks

sayakpaul requested review from a-r-r-o-w, stevhliu and yiyixuxu February 24, 2025 04:11

Merge branch 'main' into pipeline_marigold_intrinsics
b83c0a3

stevhliu approved these changes Feb 24, 2025
View reviewed changes

yiyixuxu approved these changes Feb 24, 2025
View reviewed changes

Comment threadsrc/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py Outdated

a-r-r-o-w approved these changes Feb 24, 2025
View reviewed changes

Comment threadsrc/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.py

toshasand others added 10 commits February 24, 2025 23:26

Update docs/source/en/api/pipelines/marigold.md
dc9cd47
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
d15a951
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
8972a40
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
13204ad
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
fd57911
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
149d4e9
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
2cff146
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
05c60a1
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update docs/source/en/using-diffusers/marigold_usage.md
8364655
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update marigold.md
d6ff273

toshas added 9 commits February 24, 2025 23:36

add torch.compiler.disable to progress_bar to keep it in sync with th…
2ba8d4d
…e base

make possible to instantiate the pipeline without vae and unet
d444508
compute n_targets on the go add a consistency check for vae_scale_factor

revert to having n_targets as a pipeline property
b3d1152

minor depth ensembling fixes
3ecbcd7

improve api documentation structure
9489028

attempt at fixing latex jammed into preceding words
68be852

enhance marigold usage section on acceleration
1e97e03
move quantitative evaluation and predictive uncertainty sections of marigold into the end

improve marigold video processing example
c5c5e12

improve tutorial conclusion
54f67f5

yiyixuxu merged commit 3fab662 into huggingface:mainFeb 26, 2025

toshas deleted the pipeline_marigold_intrinsics branch May 15, 2025 17:34

Uh oh!

Conversation

toshas commented Feb 23, 2025

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Feb 24, 2025

Uh oh!

toshas commented Feb 24, 2025

Uh oh!

a-r-r-o-w commented Feb 24, 2025

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

a-r-r-o-w left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

toshas commented Feb 25, 2025

Uh oh!

yiyixuxu commented Feb 26, 2025

Uh oh!

nitinmukesh commented Feb 26, 2025

Uh oh!

toshas commented May 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

a-r-r-o-w left a comment •
edited
Loading