Uh oh!
There was an error while loading. Please reload this page.
[docs] Distributed inference - #12285
Conversation
HuggingFaceDocBuilderDev
commented
Sep 4, 2025
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Uh oh!
There was an error while loading. Please reload this page.
| > [!TIP] | ||
| > You can use `device_map` within a [`DiffusionPipeline`] to distribute its model-level components on multiple devices. Refer to the [Device placement](../tutorials/inference_with_big_models#device-placement) guide to learn more. | ||
| ## Model sharding |
There was a problem hiding this comment.
I removed it because it seems more like a "recipe" for progressively and strategically fitting models on a GPU by loading and removing them. I don't think a user is really learning anything new/useful about device_map here compared to the device placement docs to which there is a link at the bottom.
I would suggest removing it or at least moving it to Resources > Task Recipes.
There was a problem hiding this comment.
Removal doesn't sound right as the content is useful IMO. Including it in "Task Recipes" might also hamper its discoverability.
b1713b2 to
90409ddCompareUh oh!
There was an error while loading. Please reload this page.
Reduces bloat by removing the
## Model shardingsection because it isn't really about distributed inference (its more selectively loading and deleting models) and doesn't show processing multiple prompts in parallel.