whisper: opt-in adaptive encoder window for short-form audio (~3x on short clips) by random1st · Pull Request #149 · handy-computer/transcribe.cpp

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st · 2026-08-29T09:34:54Z

What

Whisper's encoder always processes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence. Measured on M3 Max (Metal, large-v3-turbo Q8_0): encode cost is a flat ~275 ms per clip regardless of length. Since run_whisper_encoder_on_window already rebuilds the graph per window (it takes n_mel_frames and reallocates on T_enc change), the window can be sized to the audio instead. On 2–6 s utterances this cut end-to-end latency ~3× in our deployment (a voice assistant, where short utterances dominate).

Why opt-in, and why short-form only

Short-form only. We first tried shrinking the window globally — it re-chunks long-form and destroys it (measured: long-form Russian WER 4.31% → 85.34% at a 20 s window), because the seek/stitch path assumes the native window size. This patch touches only the is_short_form sizing; long-form is bit-for-bit unchanged.
Opt-in. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. It's gated behind TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so each deployment can price the trade. With the default 10 s floor we measured no WER regression on a 26-clip ru/en/code-switching corpus (clean + noisy).
Default behaviour unchanged. Without the env var, the window stays at the model's native size for both forms.

Validation

Shipped in production via the Rust transcribe-cpp-sys bindings (patched crate) on macOS/Metal for several weeks of daily use.
Guarded against the window ever exceeding enc_max_source_positions, and against odd frame counts.
Uses the existing transcribe::env helpers; no new dependencies.

Happy to adjust knobs/naming or move the gate into run_params if you'd prefer a non-env API.

Whisper encodes a fixed 30 s window, so a 4 s utterance pays for 26 s of silence: measured encode cost is a flat ~275 ms per clip at any length (M3 Max, Metal, large-v3-turbo Q8_0). The encoder graph is already rebuilt per window — run_whisper_encoder_on_window takes n_mel_frames and reallocates on T_enc change — so the window can be sized to the audio instead. On 2-6 s utterances this cut end-to-end latency ~3x in our deployment. Only short-form is touched, deliberately. Shrinking the window globally also re-chunks long-form, and that destroys it (measured: long-form Russian WER 4.31% -> 85.34% at a 20 s window) — the seek/stitch path assumes the native window size. Positional embeddings were trained at 30 s, so a shorter window is a real accuracy trade. The feature is therefore opt-in via TRANSCRIBE_ADAPTIVE_WINDOW, with TRANSCRIBE_WINDOW_MIN_SECS (default 10) and TRANSCRIBE_WINDOW_MARGIN_SECS (default 2) so the trade can be priced per deployment. With the default floor of 10 s we measured no WER regression on a 26-clip ru/en/code-switching corpus; clips near the window edge are protected by the margin. Default behaviour is bit-for-bit unchanged: without the env var the window stays at the model's native size for both forms.

cjpais · 2026-08-30T00:15:17Z

I'll get to reviewing this, but I'm not confident I will pull it in. I do appreciate the thought that went into this, and it's not a wide sweeping set of changes. But I generally don't like adding ENV flags as they feel like a hack, and for the most part I am okay with some latency penalty at the cost of correctness. Especially when there are so many other models to choose from for lower latency

random1st requested a review from cjpais as a code owner August 29, 2026 09:34

Uh oh!

Conversation

random1st commented Aug 29, 2026

What

Why opt-in, and why short-form only

Validation

Uh oh!

cjpais commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

random1st commented Aug 29, 2026

What

Why opt-in, and why short-form only

Validation

Uh oh!

cjpais commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

random1st commented Aug 29, 2026

What

Why opt-in, and why short-form only

Validation

Uh oh!

cjpais commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

random1st commented Aug 29, 2026

What

Why opt-in, and why short-form only

Validation

Uh oh!

cjpais commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

random1st commented Aug 29, 2026

What

Why opt-in, and why short-form only

Validation

Uh oh!

cjpais commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels