Uh oh!
There was an error while loading. Please reload this page.
Add ABot-World - #14539
Open
yiyixuxu wants to merge 7 commits into
Open
Conversation
…-5B) Integrates https://github.com/amap-cvlab/ABot-World (acvlab/ABot-World-0-5B-LF, Apache-2.0) as a modular pipeline. - ABotWorldTransformer3DModel: block-causal rollout over a rolling K/V cache (window eviction, pinned reference-token prefix, relative RoPE with periodic rebase), per-frame timesteps, keyboard-action adapter added onto the patch tokens, per-stream cross-attention cache. Bit-exact vs the reference (CPU/fp32) incl. eviction, rebase, and the real 5B weights. - scripts/convert_abot_world_to_diffusers.py: pure key renames, no surgery. VAE and umt5 are byte-identical to Wan-AI/Wan2.2-TI2V-5B and are reused from Wan-AI/Wan2.2-TI2V-5B-Diffusers; FlowMatchEulerDiscreteScheduler(shift=5.0) covers the warped DMD grid (scale_noise == the reference re-noise). - modular_pipelines/abot_world: text/image/reference encoders -> core denoise (prepare + rollout IterativePipelineBlocks over blocks k, with a nested distilled denoise loop over (i, t) and a KV-cache context update) -> decode. Streams via pipe.stream() (events per denoise step and per ~1s block) and drives interactively via loop_step, writing new actions into the state between calls. - Converted checkpoint + runnable example: YiYiXu/ABot-World-0-5B-LF-Diffusers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HuggingFaceDocBuilderDev
commented
Aug 20, 2026
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
- ABotWorldStreamingBlocks: like the default blockset, but each block is decoded to pixels inside the rollout loop (trailing-window decode with the previous block as temporal context; independent per-block decode of the causal VAE produces artifacts), so stream() yields ready frames per ~1 s block - the rollout takes a scripted `actions` list or an `action_source` callable polled once per block (return None to stop) — a live driver is just a stream() consumer whose callable returns the player's current action; with __call__ the callable combination raises, since no frames flow back between polls - set_action reads the single current `action` from state (no actions[k] indexing); `actions` becomes optional in the prepare step Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without reference images the ref encoder emits zero reference latents with a zero reference_mask, matching the reference implementation's ref-less mode; the mask is plumbed through both transformer calls in the streaming rollout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the ABot-World streaming rollout decode(z, cache=cache) keeps the causal-conv feature cache across calls so a video can be decoded chunk by chunk with results identical to a single decode. The streaming decode step carries the cache through the rollout loop instead of re-decoding a trailing window of the previous block (bit-exact vs. a full decode; 3.3 -> 2.3 s per block on H100 at 704x1280). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ock in both presets AutoencoderTinyVideo ports madebyollin's TAEHV family as one class (config per checkpoint; defaults are taew2_2 for the Wan 2.2 latent space) with a TinyVideoDecodeCache for chunk-by-chunk decode, plus scripts/convert_taehv_to_diffusers.py and a docs page. Parity against the reference: decode 1.5e-8, streaming 2.5e-6, encode exact. The ABot-World presets now mirror the reference's vae_type switch: ABotWorldBlocks decodes every block inside the rollout loop with the full Wan VAE (ABotWorldDecodeStep + WanDecodeCache, as scripts/inference.py does), ABotWorldStreamingBlocks with the tiny VAE (ABotWorldTinyDecodeStep, as the interactive space does). The one-shot end decode is removed; it mirrored an entry point the reference never uses and needed all latents decoded at once. Streaming block time on H100: 2.3 s -> 0.9 s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ist or a callable `actions` - `action` is a loop variable next to `k`: the rollout loop passes each block's vector to its sub-blocks instead of writing it into the state, so a single `ABotWorldSetActionStep` serves both presets (`ABotWorldCurrentActionStep` removed) and a live driver calls `loop_step(components, state, k=k, action=action)`. - `actions` is either a scripted list (one block per vector) or, with `stream()`, a callable `(block_index) -> vector or None` polled once per block; `action_source` and the separate streaming wrapper are gone. The presets now differ only in their decode step. - Adopt `loop_inputs` / `loop_intermediate_outputs` from the base: `actions` and `denoise_timesteps` are loop inputs, `videos` is written through `set_block_state`. - `decode_cache` is seeded as `None` by the prepare step (created by the decode step on the first block) instead of being filtered out of the loop's inputs; the prepare step no longer validates `actions` — the set-action step checks each block's vector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hub repo: https://huggingface.co/YiYiXu/ABot-World-0-5B-LF-Diffusers
space demo: https://huggingface.co/spaces/diffusers-internal-dev/abot-world-interactive
to run it iteractively