parakeet: optimize long ChunkedLimited attention

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.

[taf2]

Summary\n\n- Replace quadratic dense ChunkedLimited attention on long single-utterance Parakeet inputs with an equivalent batch of bounded query/key windows.\n- Keep the dense reference graph for short, batched, and tensor-debug runs; the bounded path starts only beyond the measured crossover.\n- Reuse the allocated steady-state cache-aware streaming graph instead of rebuilding it for every chunk.\n\nFor the Nemotron 3.5 geometry, each 14-frame query chunk attends to a 70-frame K/V window. This changes attention score work and temporary storage from O(TT) to O(TW), while preserving the original chunk topology and relative-position offsets.\n\n## Scope\n\n- src/arch/parakeet: offline graph selection, compact masks and positional embeddings, streaming graph lifecycle/replay\n- src/conformer: bounded relative-position attention using portable ggml reshape, pad, im2col, and flash/manual attention operations\n- tests: exhaustive compact-mask, im2col-window, and relative-position equivalence coverage\n\n## AI Assistance\n\nCodex meaningfully assisted with profiling, implementation, test construction, numerical comparison, and benchmark analysis. I reviewed and validated the resulting change and own the contribution.\n\n## Validation\n\n- scripts/ci/clang-format.sh --check\n- cmake --build build --target transcribe-cli\n- ctest --test-dir build --output-on-failure -j8: 34/34 passed\n- uv run scripts/validate.py cpp --family parakeet --variant nemotron-3.5-asr-streaming-0.6b --gguf models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b-F16.gguf --backend metal\n- Q8_0 and F16 Metal inference, Q8_0 CPU fallback, short/dense and long/bounded paths\n- Cache-aware streaming R=0,3,6,13: output identical to the pre-change binary\n- ctm-transcribe integration tests: 5/5 passed, including real-model Metal smoke\n\nApple M4 Max, Q8_0, 557.56-second WAV:\n\n- HTTP end-to-end: 23.54 s before, 10.21-10.90 s after (2.2-2.3x)\n- Encoder: 19.38 s before, approximately 5.3 s after (approximately 3.6x)\n- Transcript text and token count are identical.\n- Direct tensor comparison showed final encoder mean absolute drift of 2.09e-5 under Metal flash attention. One token boundary moved by 80 ms because the bounded geometry selects a different Metal reduction kernel.\n\nThe new mask unit test exhaustively compares more than 10,000 combinations of sequence length, chunk size, left context, prefix padding, ragged tails, im2col origins, and relative-position indices against the dense formula.\n\nA corpus-level WER run is still recommended because floating-point reduction order changes, even though the attention topology is exactly equivalent and the tested transcripts remain identical.