Repository navigation
test: the warm recall test runs with thinking off and bounded, on SmolLM2 and Qwen3.5-4B - #61
Merged
Merged
Conversation
…lLM2 and Qwen3.5-4B testWarmMultiTurnRecall let a thinking model think until the cache held exactly nCtx cells, at 2048 and 8192 alike, and the second turn failed with no memory slot for a batch of one. That was the test's design, not the engine's. The case now formats both turns with thinking disabled, bounds generation at 512 tokens, and runs on the text model and on Qwen3.5-4B when its file is present, in the branch suite. Green on both.
There was a problem hiding this comment.
🟢 Approval recommended
The focused test-only changes have no identified blocking issues; model-based tests were not rerun during review.
No open issues 🎉
What changed in this PR
Updates the warm multi-turn recall integration test to prevent unbounded thinking from exhausting the context cache.
Changes:
- Disables thinking for both turns and adds a 512-token generation bound.
- Tests the configured model and optional Qwen3.5-4B model, including in the
branchsuite.
| File | Description |
|---|---|
| test/integration.ts | Bounds recall-test generation and expands model and suite coverage. |
Review effort: Balanced
… suite compose; the bound is checked once Qwen3.5-4B is named once, beside the other optional models, and the list of models a text case runs on (the default, and the hybrid when its file is present) is one definition both the warm-recall and decode-failure cases read. The warm-recall bound is checked once after the loop, so a turn prints one result rather than one per token. Verified against the local build: branch suite on Qwen3.5-4B 51 passed, text suite on SmolLM2 332 passed. Gate: gets in. The hybrid model path and the default-plus-hybrid selection rule each move from several places to one, declared beside the other model constants where the file already keeps them, so the next change to which models a case runs on touches one line.
The warm recall case runs on Qwen3.5-4B as well as SmolLM2, and on the macOS runner that case alone takes about eight minutes, so the integration step hit its ten-minute limit on every head since the case was added. The step that runs the text suite has thirty minutes, as the image step that runs the same model has forty-five.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLO-40:
testWarmMultiTurnRecallfailed on Qwen3.5-4B at the second turn with "failed to find a memory slot for a batch of one", at 2048 and 8192 cells alike, on b9581 and b11559 alike. Reproduced againstbuild/Releasewith the decode log on: the test let a thinking model think with no bound, so the first turn filled the cache to exactlynCtxcells and the second turn had nowhere to go. The engine and the addon did what they should; the test's design was the cause. Nothing to file upstream, and no engine change.The case now formats both turns with
enableThinking: false, bounds generation at 512 tokens and asserts the bound, and runs on the suite's text model and on Qwen3.5-4B when../models/Qwen3.5-4B-Q4_K_M.ggufis present, as part of thebranchsuite.Verified against the local build:
--suite=branchon Qwen3.5-4B, all passed;--suite=texton SmolLM2, all passed.Release: lloyal.node 4.1.1 is a pin bump to liblloyal 3.1.1 (lloyal-ai/liblloyal
feature/run-fixes: an absent seed keeps the chain's seed) plus this test. The submodule moves to the 3.1.1 merge commit at the release.