Shap-E: add support for mesh output by yiyixuxu · Pull Request #4062 · huggingface/diffusers

yiyixuxu · 2023-07-12T14:38:53Z

This PR adds support to mesh output for Shap-E
#3986

importtorchfromdiffusersimportDiffusionPipelinefromdiffusers.utilsimportexport_to_plydevice=torch.device('cuda'iftorch.cuda.is_available() else'cpu')
#repo_prefix = "openai"repo_prefix="YiYiXu"pipe=DiffusionPipeline.from_pretrained(f"{repo_prefix}/shap-e", torch_dtype=torch.float16)
pipe=pipe.to(device)
guidance_scale=15.0prompt="a shark"generator=torch.Generator(device=device).manual_seed(0)
images=pipe(
prompt, generator=generator, guidance_scale=guidance_scale,
num_inference_steps=64, output_type="mesh"
).imagesply_path=export_to_ply(images[0], "3d_shark.ply")
print(f"saved to folder: {ply_path}")

HuggingFaceDocBuilderDev · 2023-07-12T14:45:54Z

The documentation is not available anymore as the PR was closed or merged.

yiyixuxu · 2023-07-12T15:09:47Z

+ for k, (c1, c2) in enumerate(zip(tri[::2], tri[1::2])):
+ cases[i, j, k] = edge_to_index[(c1, c2) if c1 < c2 else (c2, c1)]
+ masks[i, j] = True
+ return cases, masks


I saved this lookup table as part of weights for renderer but this is how we can generate it if we want to include it in the code instead
cases is tensor of shape 256 x 5 x 3, masks is 256 x 5

patrickvonplaten

Great! very clean! Can we maybe just add a new section to the docs, showing how one can generate a mesh (just change output type) and then maybe linking to a generated mesh?

yiyixuxu · 2023-07-13T06:02:34Z

@patrickvonplaten

linking to a generated mesh

What does this mean?

sayakpaul · 2023-07-13T06:04:14Z

What does this mean?

I think Patrick meant hosting the generated mesh file on a repository on Hub so that folks can use that. If there are online visualization tools it would be cool to host one such generated mesh file and leave a link to the instance.

@hysts what do you suggest?

sayakpaul · 2023-07-13T06:08:13Z

I think @yiyixuxu has updated the Shap-E repository from her personal profile. Let's not forget to transfer them to OpenAI and update the model cards accordingly with this support.

hysts · 2023-07-13T07:20:53Z

I think Patrick meant hosting the generated mesh file on a repository on Hub so that folks can use that. If there are online visualization tools it would be cool to host one such generated mesh file and leave a link to the instance.
@hysts what do you suggest?

As HF Hub supports mesh visualization, maybe we can upload generated 3D mesh samples as dataset.

out.mp4

hysts

Awesome! LGTM!

sayakpaul · 2023-07-13T07:45:38Z

#4062 (comment)

@yiyixuxu let's make sure this in the documentation :)

hysts · 2023-07-13T07:59:06Z

FYI, as a minor point, the output mesh of the Shap-E model is from the bottom viewpoint, so it might be useful if export_to_ply and export_to_obj had an option to rotate the model.

The default viewpoint is like this:

and I rotate this mesh like this:

importtrimeshmesh=trimesh.load('3d_shark.obj')
rot=trimesh.transformations.rotation_matrix(-np.pi/2, [1, 0, 0])
mesh=mesh.apply_transform(rot)
mesh.export('3d_shark2.obj', file_type='obj')

Also, it might be useful if .glb was supported because only .glb seems to be supported by the mesh visualization in the Hub.

But users can do these with a few lines of code, so maybe it doesn't have to be supported by the diffusers util function.

pcuenca

Very nice!

yiyixuxu · 2023-07-13T19:48:05Z

@hysts

FYI, as a minor point, the output mesh of the Shap-E model is from the bottom viewpoint, so it might be useful if export_to_ply and export_to_obj had an option to rotate the model.

great point! but this requires installing trimesh so I prefer to just add a section in documentation about how to rotate the mesh

Also, it might be useful if .glb was supported because only .glb seems to be supported by the mesh visualization in the Hub.

same, I think it requires additional dependency so prefer to document it instead

CC @patrickvonplaten let me know what you think about this

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

yiyixuxu · 2023-07-13T21:49:48Z

As HF Hub supports mesh visualization, maybe we can upload generated 3D mesh samples as dataset.

this is so cool! thanks! I didn't know that
https://huggingface.co/datasets/hf-internal-testing/diffusers-images/blob/main/shap_e/3d_cake.glb

sayakpaul · 2023-07-14T03:33:56Z

great point! but this requires installing trimesh so I prefer to just add a section in documentation about how to rotate the mesh

same, I think it requires additional dependency so prefer to document it instead

Big +1 here. Let's make sure to document them. And also let the readers know about the capability of https://huggingface.co/datasets/hf-internal-testing/diffusers-images/blob/main/shap_e/3d_cake.glb. Together, this collection is starting to look really really amazing.

Thanks for your hard work, @yiyixuxu ❤️

sayakpaul

Amazing! 🔥

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

patrickvonplaten · 2023-07-17T15:26:34Z

Feel free to merge @yiyixuxu

yiyixuxu · 2023-07-20T01:19:04Z

@patrickvonplaten can I have a final look? think it is good to go now

sent 2 PR on the openai repo too:
https://huggingface.co/openai/shap-e/discussions/10
https://huggingface.co/openai/shap-e-img2img/discussions/7

yiyixuxu · 2023-07-20T01:25:36Z

ohh forgot about tests... fixing it now

pcuenca

The tests look fine to me. I suppose we just need to merge the PRs in the HF repos, right?

patrickvonplaten · 2023-07-20T16:05:10Z

I think it would have been better to leave renderer as is (not delete it from the init) and instead just add a new renderer_table to the init. This would have made this PR 100% backwards compatible and would not force a warning going forward, but ok for now.

yiyixuxu · 2023-07-20T18:25:05Z

think it would have been better to leave renderer as is (not delete it from the init) and instead just add a new renderer_table to the init.

ohhh yes should have done that

* add output_type=mesh * update img2img * make style * add doc * make style * Apply suggestions from code review Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * add docstring for output_type * add a section in doc about hub mesh visualization/ rotation * update conversion script so default background is white * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * renderer -> shap_e_renderer * img2img renderer -> shap_e_renderer * fix tests --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

yiyixuxu added 3 commits July 12, 2023 10:22

add output_type=mesh
fc6716a

update img2img
e9a3dd2

make style
ba9679d

yiyixuxu changed the title ~~Shap ee mesh~~Shap-E: add support for mesh output Jul 12, 2023

yiyixuxu commented Jul 12, 2023
View reviewed changes

patrickvonplaten approved these changes Jul 12, 2023
View reviewed changes

yiyixuxu requested a review from hysts July 13, 2023 05:56

sayakpaul reviewed Jul 13, 2023
View reviewed changes

Comment threadsrc/diffusers/pipelines/shap_e/renderer.py

yiyixuxu added 2 commits July 13, 2023 06:22

add doc
da049da

make style
480365e

hysts approved these changes Jul 13, 2023
View reviewed changes

pcuenca approved these changes Jul 13, 2023
View reviewed changes

Comment threaddocs/source/en/api/pipelines/shap_e.mdx Outdated
Comment threaddocs/source/en/api/pipelines/shap_e.mdx Outdated
Comment threadsrc/diffusers/pipelines/shap_e/pipeline_shap_e.py
Comment threadsrc/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py

yiyixuxuand others added 2 commits July 13, 2023 10:50

Apply suggestions from code review
aa95d57
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

add docstring for output_type
0900336

yiyixuxu added 3 commits July 13, 2023 22:26

add a section in doc about hub mesh visualization/ rotation
77b5f98

Merge remote-tracking branch 'origin/main' into shap-ee-mesh
b33fc95

update conversion script so default background is white
8c30f88

sayakpaul reviewed Jul 14, 2023
View reviewed changes

Comment threaddocs/source/en/api/pipelines/shap_e.mdx Outdated

sayakpaul reviewed Jul 14, 2023
View reviewed changes

Comment threadsrc/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py Outdated

sayakpaul reviewed Jul 14, 2023
View reviewed changes

Comment threadsrc/diffusers/pipelines/shap_e/pipeline_shap_e.py Outdated

sayakpaul approved these changes Jul 14, 2023
View reviewed changes

yiyixuxuand others added 2 commits July 15, 2023 20:24

Apply suggestions from code review
b7c4003
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Update src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py
7e27a5f
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

yiyixuxu added 2 commits July 20, 2023 00:42

renderer -> shap_e_renderer
2ae53e9

img2img renderer -> shap_e_renderer
33a5aec

fix tests
fb21ecb

pcuenca approved these changes Jul 20, 2023
View reviewed changes

patrickvonplaten merged commit 47b3346 into mainJul 20, 2023

patrickvonplaten deleted the shap-ee-mesh branch July 20, 2023 16:06

hlky mentioned this pull request Apr 29, 2026
shap_e model/pipeline review #13593
Open

Uh oh!

Conversation

yiyixuxu commented Jul 12, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Jul 12, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxuJul 12, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu commented Jul 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Jul 13, 2023

Uh oh!

sayakpaul commented Jul 13, 2023

Uh oh!

hysts commented Jul 13, 2023

Uh oh!

hysts left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jul 13, 2023

Uh oh!

hysts commented Jul 13, 2023

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yiyixuxu commented Jul 13, 2023

Uh oh!

yiyixuxu commented Jul 13, 2023

Uh oh!

sayakpaul commented Jul 14, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Jul 17, 2023

Uh oh!

yiyixuxu commented Jul 20, 2023

Uh oh!

yiyixuxu commented Jul 20, 2023

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Jul 20, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HuggingFaceDocBuilderDev commented Jul 12, 2023•
edited
Loading

yiyixuxuJul 12, 2023•
edited
Loading

yiyixuxu commented Jul 13, 2023•
edited
Loading

patrickvonplaten commented Jul 20, 2023•
edited
Loading