Repository navigation
feat(embedding): encodeMultimodal for EmbeddingGemma 2, and a dispose that waits for work in flight - #63
Merged
Conversation
…nst llama.cpp EmbeddingGemma 2 joins the services matrix as a pinned, checksummed model and projector pair, with its own suite (`npm run test:embedding`) and CI step. The suite checks that the projector carries a vision and an audio encoder, that getEmbeddingDimension() is the model's output width, and that each query in liblloyal's reference fixture embeds within a cosine of 0.999 of the vector llama.cpp's server gives. Red on the addon built from liblloyal 3.1.1: the width read 512, the backbone's, and so did every vector. Green on liblloyal's feature/video-ability, which reads the output width. The submodule pin moves to liblloyal 3.2.0 when it is tagged; until then CI has neither the width nor the checksum manifest. The check that a suite loaded this checkout's addon moves into test/local-addon.ts, shared by the audio and embedding suites. Judged by hickey 0.2.1: Gets in: The whole makes more sense with this change. EmbeddingGemma 2 enters through the existing `services` matrix extension point, and the rule that a suite must prove it loaded this checkout's addon now has one owner (`test/local-addon.ts`) that both service suites compose. The one dependency to confirm is that the liblloyal submodule pin includes the new checksum manifest and reference fixture.
… a video window An embedding context with a projector takes a prompt with media markers and its typed media, as a branch prefill does, and embeds it whole: EncodeMultimodalWorker marshals the input into liblloyal's MtmdSource with the model's own special tokens and hands it to embedding::encode, which places the stream in one dispatch and refuses, before anything runs, one that does not fit. Read the vector with getEmbeddings(). The context remembers it was created with embeddings on, and _cellsMultimodal frames an input on such a context the same way, so the measure is exactly what that dispatch must hold. Red on today's addon: there was no encodeMultimodal. Green on liblloyal's feature/video-ability against its EmbeddingGemma 2 reference: each of the four images, the text around an image and the clip within a cosine of 0.999 of llama.cpp's vector; each query finds its own image or clip, 5 of 5; a context one entry short of the measure refuses the input and still embeds as the reference does, and one of exactly the measure embeds it. Measuring without the special tokens fails that case. Guards, each in its own process with LLOYAL_LOCAL=1 on Metal: the text suite with nomic (332 passed), test:images (52 passed), test:audio (7 passed). The suite leaves the type gate with the audio suite, for the same reason: the SDK that declares video and encodeMultimodal ships from hdk after this addon. Judged by hickey 0.2.1: Gets in: The whole makes more sense with this change: multimodal embedding is added as one more marshaller over liblloyal's `MtmdSource` and `embedding::encode`, and the special-token framing is decided once per context, so the measure and the encode stay consistent without either re-deciding the rule.
… runner The clip and the infographic are each over a thousand cells, and on a CPU each takes minutes to encode: llama.cpp's own server took about ten minutes on the CPU for the whole reference. The step gets thirty minutes, as the text suite's does. Judged by hickey 0.2.1: Gets in: The whole makes more sense with this change: the embedding suite's time budget stays a single per-step decision next to the suite it governs, now sized to its real CPU cost and explained like the text suite's.
…d parts, task framings, languages, budgets, a window The suite follows liblloyal's reference, where every entry is ordered parts (text, or an image, the clip or a recording). One media table and inputOf turn parts into encodeMultimodal's prompt and inputs; the shared setup moves into test/embedding-fixtures.ts. The cases: - every entry embeds where llama.cpp's server puts it, at a cosine of 0.999: text in each task framing and in Spanish and Japanese, images at the projector's own budget and at 70 and 1120 image tokens, the clip, the recording, and two mixed inputs; - each query finds its own image, clip or recording, in any language; - a window from the middle of the clip, at its own times, is what the clip's query finds over every still; - a mixed input costs exactly its parts and the model's framing, by _cellsMultimodal; - the image budget sets what an image costs; - the measure is exactly what one dispatch admits (as before); - a recording over its limits, and a prompt whose markers do not match its inputs, are refused, and the context then embeds as it did; - encodeMultimodal is refused on a context without a projector or not made for embeddings. Every short-lived context is freed when its use ends (withContext), and embeds on the shared context run one after another, since each clears, encodes and reads the same context. These are guards: the addon needed no change for them. Green on Metal with LLOYAL_LOCAL=1, 11 cases, on the build that also carries the next commit's dispose fix. A budget outside the projector's trained range is not refused here: llama.cpp takes a custom budget as given and exposes no range to check against, so the bound belongs to rig's catalog row. Judged by hickey 0.2.1: Gets in: The whole makes more sense with this change. Every entry is read as ordered parts through one table, and every context lives exactly as long as the scope that uses it. The next media kind is one table row, and the next short-lived context frees its GPU memory when its use ends.
…only once its work in flight completes dispose() freed the projector, the context and the model at once, while a worker of any async call could still be using them on the native pool, and ~SessionContext did the same if the object was collected mid-flight. Every worker a context queues now derives from ContextWorker. Its SessionContext::InFlight hold counts the work and keeps the JS object from being collected until the worker is destroyed, after OnOK or OnError on the JS thread. dispose() refuses every later call at once and frees the native state when the last hold is released, or immediately if none is held. It returns a Promise, resolved once the state is freed; every call returns the same one. Teardown has one owner, freeNative(): the branch store drains while the context is alive, then the projector, the context and the model. The destructor now drains the store as dispose() always did. Red first, in a child process (test/__dispose-in-flight-child.ts), because the failure is a crash: an encodeMultimodal on the infographic and the clip, disposed at once, ran on freed memory until the process's peak memory footprint reached 81 GB and the system killed it after 44 s. Green: the encode resolves, then the memory is freed, in that order. A second dispose() returns the same Promise, later calls are refused, and a context whose last reference is dropped mid-encode and collected settles its encode. Guards on Metal with LLOYAL_LOCAL=1, each in its own process: unit, the text suite with nomic (332), branch (59), images (52), audio (7), the stderr canary, and the embedding suite (12). The SDK declares dispose(): void, and it ships from hdk; its type follows in the hdk layer. Judged by hickey 0.2.1: Gets in: The whole makes more sense with this change: SessionContext now owns the single decision of when native state is freed (InFlight plus freeNative), every worker composes that decision through ContextWorker, and two teardown copies that disagreed become one.
…digest npm run test:scenarios runs EmbeddingGemma 2 over three videos the media pipeline ingested, each read from an OCI layout by its root digest with every blob verified (test/scenario-pack.ts), against ground truth a person confirmed by watching (test/scenarios/*.json: time ranges, labels and our own descriptions, no footage and no dialogue). - Every cat in a ten-minute pet compilation: 615 frames as images. The first 20 clips for "a cat" and the first 10 for "a dog" all show that animal; at least 0.9 and 0.8 of each animal's clips come before any other; a 128-wide scan rescored at 768 on the better half keeps 95% of the full scan's clips. - A car in half an hour of dash-cam footage: the scene's description puts the moment in the first three of 1,863 frames and of 232 eight-second windows, at 768, 256 and 128 wide. - Hindi speech found by an English question from its sound alone, among all 528 soundtrack windows of the three videos, English speech and music included; fused with the Hindi transcript by reciprocal rank, each moment comes first. The embed step (clear, encode, read) and the rule that embeds on one context run one at a time move to embedding-fixtures.ts, which both embedding suites now compose, with dot. Guards on existing capabilities, not red first. On Metal (M2 Pro, LLOYAL_LOCAL=1): scenarios 7 of 7 in 628 s, embedding 12 of 12. Judged by hickey 0.2.1: Gets in: The change moves the clear-encode-read embedding sequence and the one-at-a-time rule into embedding-fixtures.ts, so both suites share them instead of copying them, and keeps pack reading and ground truth with their own owners. The next suite, or the next change to how vectors are read, costs one edit.
…er seam The scenario packs were rebuilt through hdk's ClipDecoder seam, so their frames and soundtrack windows are what a page's ingest produces: frames stamped with the time each was shown (the 29.97 fps pet compilation's at 968, 1969, 2970 ms), the soundtrack mixed and decimated by media's own procedure. Each scenario pins its new root. The pet compilation's ground truth splits one clip at a cut scene detection had missed, at 120.554 s: a cat at a window, then a dog. The dog query ranked a frame after the cut seventh, which the old label counted as a cat; the frames show a dog. On Metal (M2 Pro, LLOYAL_LOCAL=1): the dash-cam and Hindi scenarios pass on the new packs, and the pet compilation's three pass after the split. Judged by hickey 0.2.1: Gets in: The change edits data only within the design already in place: each scenario's truth file is the single owner of its pack pin and its source-time labels, and the edits stay there. The whole is as coherent as it was, and the ground truth is more correct.
The dash-cam scenario now ends where its question does. After the scene's
description finds the moment, a second query within it ("the number plate
of a red SUV") ranks the one frame the plate can be read in first, and
Qwen3.5-4B with its vision projector reads it: 01-ZAM, exactly. Asked of a
frame where the plate is a few pixels wide, it answers "unreadable": the
question gives it that way out, because asked without one it invents a
plate (six confident, wrong plates across the moment's distant frames).
The legible and illegible frames are recorded in the ground truth.
The vision model pair has one owner, test/vision-fixtures.ts, which the
integration suite's image tests now use too, and withContext moves to
local-addon.ts, typed over any context that disposes, so an embedding and
a vision context share one rule for freeing GPU memory.
On Metal (M2 Pro, LLOYAL_LOCAL=1): the dash-cam scenario 4 of 4 (510 s),
the embedding suite 12 of 12, the image suite 52 of 52 with
LLOYAL_VL_STRICT=1.
Judged by hickey 0.2.1:
Gets in: The change gives the vision model pair one owner
(vision-fixtures.ts) and the context lifetime one generic owner
(withContext in local-addon.ts), then builds the plate-reading scenario
on top of both. The next vision suite or model swap therefore touches
one place instead of copying pickVlPair or a vision-only withContext.
… as llama.cpp's server embeds it)
There was a problem hiding this comment.
🟡 Changes recommended
Disposal from a media getter can clear native handles before worker construction, risking a process crash.
1 open issue
What changed in this PR
Adds multimodal embedding support for EmbeddingGemma 2 to the native addon and defers context cleanup until in-flight work completes.
Changes:
- Adds
encodeMultimodaland embedding-aware input measurement. - Gives context workers lifetime holds and makes
dispose()return a shared completion promise. - Adds reference-vector, disposal, and media-scenario tests, plus CI integration.
| File | Description |
|---|---|
tsconfig.test.json |
Excludes suites awaiting updated SDK types. |
test/vision-fixtures.ts |
Centralizes vision model paths. |
test/scenarios/hindi-movie-dialog.json |
Defines Hindi speech retrieval ground truth. |
test/scenarios/dash-cam-crashes.json |
Defines scene and plate-reading targets. |
test/scenarios/cats-and-dogs.json |
Labels pet clips for retrieval checks. |
test/scenario-pack.ts |
Loads digest-verified media packs. |
test/scenario-integration.ts |
Tests retrieval across long media scenarios. |
test/matrix.json |
Adds the embedding model and projector. |
test/local-addon.ts |
Shares local-addon verification and cleanup helpers. |
test/integration.ts |
Uses shared vision model paths. |
test/embedding-integration.ts |
Tests embeddings, validation, and disposal. |
test/embedding-fixtures.ts |
Provides embedding contexts and media helpers. |
test/audio-integration.ts |
Reuses local-addon verification. |
test/__dispose-in-flight-child.ts |
Isolates in-flight disposal and collection checks. |
src/SessionContext.hpp |
Declares multimodal encoding and lifecycle state. |
src/SessionContext.cpp |
Implements encoding and deferred native cleanup. |
scripts/download-test-models.sh |
Documents embedding service downloads. |
README.md |
Describes embedding integration coverage. |
package.json |
Adds embedding and scenario test commands. |
.github/workflows/tests.yml |
Runs embedding integration with a CPU timeout. |
Review effort: Balanced
…han reading past an empty table)
…t disposes it cannot free it underneath Reading a caller's object runs its getters, and a getter can call dispose(). With no work in flight, dispose() freed the projector, the context and the model at once, and the call that was reading its arguments went on to use them. encodeMultimodal, _cellsMultimodal and _storePrefillMultimodal crashed on the freed model; formatChat and formatChatSync failed llama.cpp's model assertion; _scoreGroup crashed on the freed context; and _branchSetSamplerParams, _branchSetLogitBias, _branchSteer and _branchSetGrammarLazy aborted on the drained store's handle error. _branchPrefill, _storePrefill and _storeCommit were refused with an invalid handle, because the drained store frees every slot before their workers look. Every method but dispose() is now registered through held<Method>, which takes the InFlight hold the workers already take for the span of the call. A call that began before dispose() is in flight like any queued work: dispose() refuses every later call, and the native state is freed when the last hold is released, after the call and any work it queued. Freeing keeps its one owner, InFlight and freeNative(). Red first. A getter on an argument disposes the context during the call: - EmbeddingGemma 2, test:embedding: encodeMultimodal and _cellsMultimodal reading an input, and encodeMultimodal reading its audio limits. The process exited 139. - Qwen3.5-4B with its projector, the image suite: _storePrefillMultimodal reading an input, and reading a prompt. The process exited 139 after 52 passes. - SmolLM2, the text suite: formatChat, formatChatSync, _scoreGroup, the four branch setters, _branchPrefill, _storePrefill and _storeCommit. The process exited 134 on the first case, at GGML_ASSERT(model != nullptr). Green: every call settles, then the context is freed, in that order, and a later call is refused as disposed. Verified on Metal with LLOYAL_LOCAL=1, each suite in its own process: the embedding suite 13 of 13, images 54, the text suite with SmolLM2 342, branch 59, audio on Qwen3-ASR 7 of 7, unit, and the stderr canary. Cost: the hold takes and releases a reference per call. isStopToken in a tight loop went from about 72 ns to about 116 ns per call. The four branch setters still abort the process for any invalid handle, disposed or not: their std::runtime_error is not translated to a JS error. Gate: Gets in: The change puts the decision 'a running call holds the context' with the existing InFlight hold and applies it once per entry point in the registration table, which closes a use-after-free without adding a second freeing rule. The only cost: a new method has to be registered through `held<>`.
lefthook runs `hickey committing` before each commit, writes its verdict into the commit's trailers and records it after, as in hdk. The judge reads hickey's own principles; `kind` is framework, and judging is required. `npm install` installs the hooks through `prepare`. The lockfile gains only hickey, lefthook and zod. Its stubs for this version's platform packages are left as the release workflow keeps them. Gate: gets in. It gets in: commit-time judgment is placed in the git hook lifecycle, with lefthook owning the wiring, hickey owning the verdict and one config file owning the policy, so the next change to either the policy or the hooks touches one file. The only follow-ups are documentation: CONTRIBUTING.md doesn't mention the hook, and the comment cites a PRINCIPLES.md this repository doesn't contain. Refused-Before: 3 refusals on the way to this shape. Last: The rule 'freed state is never touched by a call still running' belongs in the dispose/InFlight lifecycle, which counts users. Here it is re-decided by hand at 10 call sites while at least 11 other entry points with the same exposure are left open, so the author should extend InFlight to cover a synchronous entry point instead of adding re-checks one line at a time.
Gate: gets in. The gate still has one owner (lefthook.yml, installed by prepare), and this change describes it accurately in the contributor guide while removing a reference to a file that doesn't exist, so the whole system makes more sense with it.
… no hooks to install `prepare` ran `lefthook install`, a dev dependency, on every install of the package's own tree. The GPU test images install it with --omit=dev and no git checkout, so their install failed with "lefthook: command not found". The hooks are now installed only where there is a checkout and lefthook to install them; anywhere else `prepare` does nothing and the install goes on. Gate: gets in. The change gets in. It gives one owner, the `prepare` script, to the decision of when commit hooks get installed. Images with no dev dependencies or no git checkout now install cleanly, and how hooks work stays with lefthook.
…opies nothing else The images copy package.json alone before they install, so a guard in scripts/ was not there to run: prepare failed with "Cannot find module" instead of "lefthook: command not found". The guard is now prepare's own command, and the test installs from the manifest alone, as the images do. Gate: gets in. The hook-install guard now lives entirely in the `prepare` lifecycle that owns it, so installs no longer depend on images copying `scripts/`. That makes the whole more coherent than today, and the only cost is the guard's explanation moving from a code comment to the test that enforces it.
…dows the whole CI's macOS build has Metal off, since the runner's paravirtual GPU computes wrong, so the projector runs on a three-core CPU: 40 s a clip frame, minutes an image, ten minutes for the infographic. The whole reference is over an hour there and the step timed out at 30 minutes, ten tests unrun. With LLOYAL_EMBED_LEAN=1, set on macOS only, the suite checks the text, the smallest image budget and the recording against the reference and skips the two search tests, which read every image and the clip; every contract test still runs. Run here on Metal: lean 11 pass, 2 skipped; whole 13 pass in 39 s. Gate: gets in. The whole makes more sense with the change: the workflow decides which runner runs lean, the suite decides what lean leaves out, and each decision has one owner joined by one env name. The author should make the reference-check test report what lean omitted instead of keeping a title that claims every input.
…reference Under LLOYAL_EMBED_LEAN the reference test checks a subset while its title names every input. It now reports, in the run's output, how many entries it checked and which it left out, and asserts the lean subset still holds an image, so a macOS pass never reads as the whole reference. Gate: gets in. The whole makes more sense with this change. The lean subset still has one owner, the `checked` filter. The test now guards that the subset keeps an image and reports what it skipped, derived from that filter, so lean runs can neither silently drop images nor hide what they skipped.
macOS's CI step timed out in the dispose child: the lean setting reached it but it ignored it, encoding the infographic and the clip twice, over two thousand cells each, on a CPU runner. The property needs an encode in flight when the context is freed or collected, not the largest one. Lean, the parent now names the lean reference's image and budget, and checks the child encoded no more than that; every lifecycle assertion stays. Red on Metal: 2407 cells against the lean image's 64. Green: the case in 4.7 s; the suite lean 11 passed, 2 skipped (search), and whole 13 of 13. Gate: gets in. The whole makes more sense with this change, because the dispose case's input now follows the one existing lean selection (`checked`/`smallestBudget`) through a narrow argv contract. The tautological `cells` re-measure in the parent and the child's `cells` field could be dropped with nothing lost.
lloyal-research
force-pushed
the
feature/video-ability
branch
from
October 26, 2026 11:38
0e0e8ce to
11b9482
Compare
Gate: gets in. This submodule bump changes only the liblloyal version, through the gitlink that owns it, and the llama.cpp invariant still has its owner and its CI check. So the whole is as coherent as before, with the newer core. One caveat: I couldn't inspect the submodule's contents, so if .llama-cpp-version moved, the llama.cpp gitlink has to move in the same commit.
lloyal-research
marked this pull request as ready for review
October 26, 2026 11:40
Gate: gets in. The bump changes the liblloyal version only at its single owner, the gitlink, and leaves the llama.cpp pin derived from liblloyal's `.llama-cpp-version` and checked by CI, so ownership stays as coherent as it was. The caveat is that the submodule's own changes could not be inspected, and if they moved the llama.cpp tag, the llama.cpp gitlink has to be synced before this passes CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
encodeMultimodalembeds images, text around them, audio and video windows on an embedding context, through liblloyal's segments rail; a disposed context frees its native state only once its work in flight completes.Draft: the release this belongs to still adds prefill progress (an
AsyncProgressWorkerover liblloyal's per-input progress), and the submodule moves to liblloyal's merge commit once lloyal-ai/liblloyal#52 lands.What changes
encodeMultimodal(prompt, inputs, audioLimits?)on an embedding context with a projector: one input, one pooled vector, read whole in one dispatch. Anything else is refused before any work.dispose()refuses every later call at once, frees the native state when the last call in flight completes, and returns one promise, resolved then. Every worker a context queues holds it, and a context collected mid-flight drains the same way.Verification