qwen3_asr: support larger output by jl · Pull Request #106 · handy-computer/transcribe.cpp

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl · 2026-07-22T12:33:09Z

Summary

Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).

The original code had a hard-coded limit of 256 output tokens:

constexpr int k_max_new = 256;

Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on --n-ctx or the model default (75% audio, 25% output tokens).

Scope

The only file changed is arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.

Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.

AI Assistance

No AI assistance was used to write, test, or comment the code.

Validation

before change: cannot fully transcribe a 2-minute audio clip (output truncated)
after change: can successfully fully transcribe a 2-minute audio clip

There should not be any change to WER or numerical validation as a result of these changes.

The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.

So that run() and run_batch() are more parallel.

cjpais · 2026-07-22T12:43:56Z

Have you considered #74?

jl · 2026-07-22T16:48:15Z

#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if max_new_tokens is not specified. The minor refactoring/cleanup aspects could also be moved to a separate PR if desired.

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

jl added 5 commits July 22, 2026 13:41

use k_max_new instead of local hard-coded value
d74321a

use k_prompt_overhead instead of local hard-coded value
f732577

rename want_n_ctx to max_n_kv
8644659
So that run() and run_batch() are more parallel.

clang-format fixes
65d4645

jl requested a review from cjpais as a code owner July 22, 2026 12:33

jl mentioned this pull request Jul 22, 2026
qwen3 asr max_new and k_max_new #95
Open

Uh oh!

Conversation

jl commented Jul 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

AI Assistance

Validation

Uh oh!

cjpais commented Jul 22, 2026

Uh oh!

jl commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jl commented Jul 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

AI Assistance

Validation

Uh oh!

cjpais commented Jul 22, 2026

Uh oh!

jl commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jl commented Jul 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

AI Assistance

Validation

Uh oh!

cjpais commented Jul 22, 2026

Uh oh!

jl commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jl commented Jul 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

AI Assistance

Validation

Uh oh!

cjpais commented Jul 22, 2026

Uh oh!

jl commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

jl commented Jul 22, 2026•
edited
Loading

jl commented Jul 22, 2026•
edited
Loading

jl commented Jul 22, 2026•
edited
Loading

jl commented Jul 22, 2026•
edited
Loading

jl commented Jul 22, 2026•
edited
Loading

jl commented Jul 22, 2026•
edited
Loading