Uh oh!
There was an error while loading. Please reload this page.
Let the tool-call opener win over a reasoning close, and treat an absent KV cache as already cleared - #30
Open
mcharytoniuk wants to merge 1 commit into
Open
Conversation
…nt KV cache as already cleared
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two regressions found while upgrading Paddler from 0.12.0 to 0.13.0. Both were bisected against 0.12 on the same machine, model files and prompts.
1. A tool-call opener that is also a reasoning close is swallowed
For Qwen3.5, 0.13 reports
reasoning_closes: [[248069], [248058]]whiletool_call_openis[248058]— the same token is in both lists, which is correct (<tool_call>really does end reasoning).try_consume_marker_at_tailprobed reasoning closes before the tool-call open, so the token was consumed as a reasoning terminator and the tool-call section never opened. Downstream the whole call was classified as content and never reachedparse_chat_message.Measured on the same prompt (283 prompt tokens, greedy sampling, byte-identical model output): 0.12 produced 25 tool-call tokens and a parsed call; 0.13 produced 0 tool-call tokens and 26 content tokens.
Probing the tool-call open first fixes it, and is the better answer inside a reasoning section too:
<tool_call>there ends reasoning and starts the call, so landing inToolCallbeats landing inContent. A close marker that is only a reasoning close is unaffected, so the existing suppression of a stray</think>in content still holds.2.
clear_kv_cachefails on models that have no KV cachenomic-embed-text-v1.5is non-causal and its context has no memory module, sollama_get_memoryreturns null. 0.12'sclear_kv_cachewas infallible and no-opped; 0.13 returnsMemoryHandleUnavailable, which broke embedding-only agents.Mutating an absent KV cache is vacuously satisfied, so
clear_kv_cache,clear_kv_cache_seqandkv_cache_seq_keepnow succeed with nothing to do.copy_kv_cache_seqstill errors, andkv_cache_seq_add/kv_cache_seq_div/kv_cache_seq_pos_maxkeep their own handling — they ask for or move data rather than assert an end state.Tests
Written first, confirmed failing, then fixed:
sampled_token_classifier: two unit tests for a token shared betweenreasoning_closesandtool_call_open, from both the content and reasoning sections.kv_cache_without_memory_module: an LLM test onnomic-embed-text-v1.5asserting the three mutations succeed.make clippy,make test.unit(884 passing) and the fullllama-cpp-bindings-testssuite (15 model phases, 0 failures) all pass on CUDA.One thing worth your call:
clear_kv_cacheandkv_cache_seq_keepcan no longer fail, so theirResultis now alwaysOk. Making them infallible would be the honest signature and would match 0.12, but it is a breaking change, so I left them as-is.