Add Wan2.2-S2V: Audio-Driven Cinematic Video Generation

[tolgacangoz]

This PR is fixing #12257.

Comparison with the original repo

When I put with torch.amp.autocast('cuda', dtype=torch.bfloat16): onto the transformer only and converted the initial noise's dtype into torch.float32 from torch.bfloat16 in the original repo, the videos seem almost the same. As far as I can see, the original repo's video has an extra blink.

wan.mp4

diffusers.mp4

Try WanSpeechToVideoPipeline!

!gitclonehttps://github.com/tolgacangoz/diffusers.git%cddiffusers#!git switch "integrations/wan2.2-s2v" # This is constantly changing...
!gitswitch"wan2.2-s2v"
!pipinstallpipuv-qU
!uvpipinstall-e".[dev]"-q
!uvpipinstallimageio-ffmpegftfydecordninjapackagingkernels-q# For Flash attention 2:#!uv pip install flash-attn --no-build-isolation# For Flash attention 3 in diffusers:#import os#os.environ["DIFFUSERS_ENABLE_HUB_KERNELS"] = "yes"importnumpyasnpimporttorch, osfromdiffusersimportAutoencoderKLWan, WanSpeechToVideoPipelinefromdiffusers.utilsimportexport_to_video, load_image, load_audio, load_videofromtransformersimportWav2Vec2ForCTCmodel_id="Wan-AI/Wan2.2-S2V-14B-Diffusers"# will be officialmodel_id="tolgacangoz/Wan2.2-S2V-14B-Diffusers"audio_encoder=Wav2Vec2ForCTC.from_pretrained(model_id, subfolder="audio_encoder", dtype=torch.float32)
vae=AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
pipe=WanSpeechToVideoPipeline.from_pretrained(
model_id, vae=vae, audio_encoder=audio_encoder, torch_dtype=torch.bfloat16,
)#.to("cuda")pipe.enable_model_cpu_offload()
#pipe.transformer.set_attention_backend("flash") # FA 2#pipe.transformer.set_attention_backend("_flash_3_hub") # FA 3first_frame=load_image("https://raw.githubusercontent.com/Wan-Video/Wan2.2/refs/heads/main/examples/i2v_input.JPG")
audio, sampling_rate=load_audio("https://github.com/Wan-Video/Wan2.2/raw/refs/heads/main/examples/talk.wav")
importmathdefget_size_less_than_area(height,
width,
target_area=1024*704,
divisor=64):
ifheight*width<=target_area:
# If the original image area is already less than or equal to the target,# no resizing is needed—just padding. Still need to ensure that the padded area doesn't exceed the target.max_upper_area=target_areamin_scale=0.1max_scale=1.0else:
# Resize to fit within the target area and then pad to multiples of `divisor`max_upper_area=target_area# Maximum allowed total pixel count after paddingd=divisor-1b=d* (height+width)
a=height*widthc=d**2-max_upper_area# Calculate scale boundaries using quadratic equationmin_scale= (-b+math.sqrt(b**2-2*a*c)) / (
2*a) # Scale when maximum padding is appliedmax_scale=math.sqrt(max_upper_area/
(height*width)) # Scale without any padding# We want to choose the largest possible scale such that the final padded area does not exceed max_upper_area# Use binary search-like iteration to find this scalefind_it=Falseforiinrange(100):
scale=max_scale- (max_scale-min_scale) *i/100new_height, new_width=int(height*scale), int(width*scale)
# Pad to make dimensions divisible by 64pad_height= (64-new_height%64) %64pad_width= (64-new_width%64) %64pad_top=pad_height//2pad_bottom=pad_height-pad_toppad_left=pad_width//2pad_right=pad_width-pad_leftpadded_height, padded_width=new_height+pad_height, new_width+pad_widthifpadded_height*padded_width<=max_upper_area:
find_it=Truebreakiffind_it:
returnpadded_height, padded_widthelse:
# Fallback: calculate target dimensions based on aspect ratio and divisor alignmentaspect_ratio=width/heighttarget_width=int(
(target_area*aspect_ratio)**0.5//divisor*divisor)
target_height=int(
(target_area/aspect_ratio)**0.5//divisor*divisor)
# Ensure the result is not larger than the original resolutioniftarget_width>=widthortarget_height>=height:
target_width=int(width//divisor*divisor)
target_height=int(height//divisor*divisor)
returntarget_height, target_widthheight, width=get_size_less_than_area(first_frame.height, first_frame.width, target_area=480*832)
prompt="Einstein singing a song."output=pipe(
image=first_frame, audio=audio, sampling_rate=sampling_rate,
prompt=prompt, height=height, width=width, num_frames_per_chunk=80,
).frames[0]
export_to_video(output, "video.mp4", fps=16)
importlogging, shutil, subprocessdefmerge_video_audio(video_path: str, audio_path: str):
""" Merge the video and audio into a new video, with the duration set to the shorter of the two, and overwrite the original video file. Parameters: video_path (str): Path to the original video file audio_path (str): Path to the audio file """# set logginglogging.basicConfig(level=logging.INFO)
# checkifnotos.path.exists(video_path):
raiseFileNotFoundError(f"video file {video_path} does not exist")
ifnotos.path.exists(audio_path):
raiseFileNotFoundError(f"audio file {audio_path} does not exist")
base, ext=os.path.splitext(video_path)
temp_output=f"{base}_temp{ext}"try:
# create ffmpeg commandcommand= [
'ffmpeg',
'-y', # overwrite'-i',
video_path,
'-i',
audio_path,
'-c:v',
'copy', # copy video stream'-c:a',
'aac', # use AAC audio encoder'-b:a',
'192k', # set audio bitrate (optional)'-map',
'0:v:0', # select the first video stream'-map',
'1:a:0', # select the first audio stream'-shortest', # choose the shortest durationtemp_output
]
# execute the commandlogging.info("Start merging video and audio...")
result=subprocess.run(
command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# check resultifresult.returncode!=0:
error_msg=f"FFmpeg execute failed: {result.stderr}"logging.error(error_msg)
raiseRuntimeError(error_msg)
shutil.move(temp_output, video_path)
logging.info(f"Merge completed, saved to {video_path}")
exceptExceptionase:
ifos.path.exists(temp_output):
os.remove(temp_output)
logging.error(f"merge_video_audio failed with error: {e}")
importrequests, tempfilefromdiffusers.utils.constantsimportDIFFUSERS_REQUEST_TIMEOUTresponse=requests.get("https://github.com/Wan-Video/Wan2.2/raw/refs/heads/main/examples/talk.wav", stream=True, timeout=DIFFUSERS_REQUEST_TIMEOUT)
withtempfile.NamedTemporaryFile(delete=False) astalk:
forchunkinresponse.iter_content(chunk_size=8192):
talk.write(chunk)
talk_file=talk.namemerge_video_audio("video.mp4", talk_file)

@yiyixuxu@sayakpaul@asomoza@dg845@stevhliu
@WanX-Video-1@Steven-SWZhang@kelseyee
@SHYuanBest@J4BEZ@okaris@xziayro-ai@teith@luke14free@lopho@arnold408

[Copilot]

Pull request overview

This PR adds support for Wan2.2-S2V (Speech-to-Video), an audio-driven cinematic video generation model. The implementation enables generating videos from a combination of text prompts, starting frames, and audio inputs, with optional pose video conditioning.

Key Changes:

• Introduces WanSpeechToVideoPipeline for audio-driven video generation
• Adds WanS2VTransformer3DModel with specialized audio encoding and injection mechanisms
• Implements audio loading utilities and video-audio merging capabilities

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
src/diffusers/pipelines/wan/pipeline_wan_s2v.py	New speech-to-video pipeline with audio conditioning and chunk-based generation
src/diffusers/models/transformers/transformer_wan_s2v.py	New transformer model with audio injection, frame packing, and causal audio encoding
src/diffusers/utils/loading_utils.py	Adds load_audio() function and extends load_video() with frame sampling
src/diffusers/utils/export_utils.py	Adds export_to_merged_video_audio() for ffmpeg-based audio-video merging
src/diffusers/audio_processor.py	New audio input type definitions and validation utilities
src/diffusers/image_processor.py	Adds resize_min_center_crop mode for Wan2.2-S2V preprocessing
src/diffusers/video_processor.py	Extends preprocess_video() with resize_mode parameter
tests/pipelines/wan/test_wan_speech_to_video.py	Comprehensive test suite for the new pipeline
tests/quantization/gguf/test_gguf.py	GGUF quantization tests for S2V transformer
scripts/convert_wan_to_diffusers.py	Conversion script updates for S2V model weights
docs/source/en/api/pipelines/wan.md	Documentation and usage examples for Wan-S2V

Comments suppressed due to low confidence (1)

src/diffusers/utils/loading_utils.py:221

• This assignment assigns a variable to itself.

 audio = audio

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

When audio is a numpy.ndarray, the assignment audio = audio is redundant and the sample_rate assumption of 16000 may be incorrect. The function signature suggests that audio can be passed as a numpy array, but there's no way to specify its actual sample rate. Consider either:

1. Requiring the sample rate as a parameter when passing numpy arrays
2. Raising an error for numpy array input without sample rate information
3. Documenting clearly that numpy arrays are assumed to be at 16000 Hz

[Copilot]

The variable src_w is used in the paste_center crop_type (line 487), but it's only defined when resize_type == "fit_within" (line 472). When resize_type == "min_dimension", src_w will be undefined, causing a NameError. You need to define src_w and src_h for the "min_dimension" resize_type as well.

[Copilot]

Test is always true, because of this condition.

Suggested change

	clean_latents_post=clean_latents_post[:, :0]ifadd_last_motion<2elseclean_latents_post
	clean_latents_post=clean_latents_post[:, :0]

[tolgacangoz]

Gentle nudge-any particular reason why this PR has not been reviewed and merged?

@yiyixuxu@sayakpaul@asomoza@dg845@stevhliu

[yiyixuxu]

hi @Tolga

I actually did go through the PR! It's a a pretty complex integration and it still requires significant refactoring to meet our standard. Unfortunately we don't have the bandwidth right now to take on the remaining refactoring ourselves.

[tolgacangoz]

FYI: Downloads last month 15,748 and 11. at Image-to-Video category in terms of most downloads.

[tolgacangoz]

As someone who began working on this PR as soon as the requesting issue was published by mentioning it as the priority, I must emphasize that this PR has been waiting for review for months. I request respect for the contributors!

@yiyixuxu@sayakpaul@DN6@asomoza@dg845@stevhliu

[yiyixuxu]

Hi @tolgacangoz,
Thanks for the effort on this PR. After reviewing, we've decided not to move forward with further iterations as it would require a significant rewrite to meet our standards.
If we find bandwidth to revisit this approach in the future, we'll make sure you're credited as co-author for the original contribution — but there's nothing more needed from your end at this point.
Closing for now.

[yiyixuxu]

hi @tolgacangoz

I also want to acknowledge - if the overall experience has felt like your time wasn't respected, that's valid feedback and something we're actively working to improve. We've learned from situations like this and are being more intentional about how we engage with contributors.
For example, in our new MVP program, we've introduced clearer guidelines: one contributor per PR at a time, starting with smaller scoped issues, and requiring a proposal before diving into implementation. This helps ensure contributors can have meaningful, successful contributions without investing months into something that may not land.

Thanks for your understanding.