Repository navigation
memory: reduce repeated topic curation with frozen preparation - #3282
Merged
Hillary Mutisya (hillary-mutisya) merged 2 commits intoNov 11, 2026
Merged
Conversation
Add opt-in prior deltas, compact wire IDs and two-worker frozen preparation with serialized, validated reconciliation and ordered watermark receipts. Harden retained-input replay with model bindings, absolute deadlines, validated partial progress and the corrected diagnostic organization handoff. Preserve default behavior and native receipt validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep deadline cancellation artifacts and persistence failures unchanged while bringing the replay lifecycle within the committed-branch complexity budget. Validation: full workspace build and all 43 replay/options tests pass; the 169 service tests passed on the unchanged service implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hillary Mutisya (hillary-mutisya)
added a commit
that referenced
this pull request
Nov 13, 2026
## Summary - Make frozen two-packet preparation the only generative topic-curation schedule, followed by ordered reconciliation and complete-window commits. - Remove the legacy serial/speculative scheduler and its configuration selectors. - Align production, replay, watermark and benchmark identities with the new defaults while preserving native Decision execution. ## Motivation #3282 added frozen preparation, but ordinary production configuration still selected the old serial path. This promotes the frozen implementation instead of retaining competing schedules that can repeatedly invalidate speculative proposals. ## Behavior and migration - Prior deltas and compact wire IDs now default to `true`. Input-token targets and reserved prior fractions remain optional; this does **not** make the historical 24,000-token / 0.25-fraction treatment the default. - Remove `TYPEAGENT_MEMORY_TOPIC_CONCURRENCY` and `TYPEAGENT_MEMORY_TOPIC_FROZEN_PREPARATION`, including their former candidate values. Their programmatic counterparts, `concurrency` and `frozenPreparation`, also fail explicitly. - Remove replay flags `--concurrency`, `--frozen-preparation` and `--no-frozen-preparation`. - Remove the `topic-parallel` experiment. `indexing-parallel` now selects only synopsis and embedding treatments; indexing ablation no longer offers the obsolete `T1` arm. - The new `curated-lazy-v10:<hash>` recipe prevents older-policy progress/receipts from qualifying as current work. Recorded replay requires a matching recording identity; historical progress is rejected rather than relabeled. - Reuse and private watermarks remain opt-in. Failed/cancelled preparation cannot checkpoint a partial window; exact validated prefixes can still be reused. - Serial reconciliation and installation are intentional. The shared single-pass evidence primitive remains necessary inside frozen preparation/reconciliation and native uncertainty resolution; it is not an alternate host scheduling mode. - Native Decision concurrency and its existing resolution behavior remain unchanged. The explicit raw diagnostic path also remains available. ## Reviewer guide 1. [Settings and policy identity](ts/packages/memory/service/src/topicCurationSettings.ts), then [generative orchestration](ts/packages/memory/service/src/topicCuration.ts). 2. [Frozen windows](ts/packages/memory/service/src/topicFrozenPreparation.ts) and [native uncertainty resolution](ts/packages/memory/service/src/topicDecisionCuration.ts). Evidence validation continues to reuse the snapshot-local index. 3. [Service wiring](ts/packages/memory/service/src/fileMemoryService.ts), [host configuration](ts/packages/memory/service/src/viewHostOptions.ts), and [captured-work reuse](ts/packages/memory/service/src/topicCapture.ts). 4. [Replay/migration guide](ts/tools/memory-maintenance-qualification/topic-curation-README.md), replay identity/capture tests, and service tests for window draining, cancellation, resumptions, append/correction/forget churn and exact output parity. ## Validation Initial promotion validated at `fc54c3cee` on Windows against `origin/main` at `3317b2a4a`. | Check | Result | | --- | --- | | `pnpm run build` | Passed, 104 tasks | | All topic service suites plus host options, maintenance priority and performance experiments | 321 passed, 17 suites | | `pnpm run test:memory-quality` | 375 passed, no skips | | `pnpm run ci:node --test tools/memory-maintenance-qualification/topic-work-benchmark.test.mjs` | 3 passed; exact frozen call counts, prefix reuse and output parity | | Repository-pinned Prettier and `git diff --check` | Passed | | Committed-HEAD lint, complexity, circular-dependency and test-debt gates against `origin/main` | All four passed; no new violations, cycles or skips | Service test selection, from `ts`, in PowerShell: ```powershell Set-Location packages\memory\service $tests = @(Get-ChildItem dist\test\topic*.spec.js | ForEach-Object { $_.FullName }) $tests += @('viewHostOptions', 'maintenanceBuildPriority', 'memoryPerformanceExperiments') | ForEach-Object { (Resolve-Path "dist\test\$_.spec.js").Path } pnpm run jest-esm --runTestsByPath @tests ``` Additional legacy offline-harness suite: **9 passed, 1 pre-existing failure**. Its worker-bootstrap test fails before the intended missing-profile check because unchanged trace instrumentation expects the absent `runner.verifyContributionRefresh` hook. Running the original `origin/main` harness assertions reproduces the same failure (8 passed, 1 failed); this is not a clean-baseline full-build claim. The newly added compiled-settings provenance assertion passes. This unrelated harness issue is not repaired here. ## CI repair at `c8b1f8aa3` The Windows Node 24 remainder shard failed in browser `viewLifecycle.test.mjs`: the real child view server exceeded its existing 10-second readiness deadline. It eagerly loaded the entire memory runtime twice through independent import paths: experiment configuration in the gateway and paging constants in the memory client. The follow-up exposes lightweight `performance-experiments` and `view-detail` service entry points and uses them at those boundaries. Existing root exports and configuration behavior are preserved. A real ESM loader-hook regression rejects accidental imports of the full memory, indexing or model runtimes during gateway startup. It failed before the repair and passes afterward. The production readiness deadline and lifecycle-test timeout are **unchanged**; no workflow, retry budget or test coverage was weakened. Local validation of the repair on Windows, Node 26.13.0: - `pnpm run build`: passed across the full workspace. - Browser `pnpm run test:local`: **1,071 passed**, including all 130 gateway tests and all 516 view-client tests. - Memory client `pnpm run test:local`: **81 passed**, including pagination/protocol coverage. - Memory service host/configuration suites: **33 passed**. - Repository-pinned formatting, whitespace check and all four committed-HEAD ratchets: passed. The new CI run must verify the originally failing Windows Node 24 shard; local validation is not a claim that the remote run has completed. ## Risks and follow-up - No new live inference or latency/semantic-quality qualification was run for this promotion. Synthetic fixtures establish scheduling, accounting and reuse invariants, not semantic equivalence or production speedup. - Frozen preparation introduces a preparation/reconciliation boundary and can require extra calls for small or all-deferred inputs. Historical candidate timings are not measurements of these default packet budgets. - Existing deployments with removed selectors must clear them. Older-policy curation may need rebuilding. - Rollback is a code revert, not a legacy scheduling flag. No automatic merge is requested. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
Serial topic curation repeatedly sends and rewrites already-admitted concepts. Speculative preparation can also become invalid when earlier decisions change its prior-concept frontier. Prepare independent raw packets against frozen inputs instead, then reconcile their proposals against current admitted concepts before installing a complete window.
Behavior and configuration
All three new treatments are off by default:
TYPEAGENT_MEMORY_TOPIC_PRIOR_DELTASTYPEAGENT_MEMORY_TOPIC_COMPACT_IDSTYPEAGENT_MEMORY_TOPIC_FROZEN_PREPARATIONThese are also programmatic
topicCurationsettings and replay CLI switches. Effective treatment settings participate in policy/replay identity. Watermark continuation accepts explicit frozen two-worker execution, not ordinary speculative concurrency or the native Decision engine. Source correction/removal, forgetting, eligibility, and current-publication checks remain fail-closed.Automatic maintenance may recapture a changed topic head once during admission; repeated churn still fails stale. Explicit callers retain strict expected-head semantics.
The replay organization handoff drops captured native packet-receipt assertions from its cloned diagnostic input. It does not disable production receipt validation or certify native watermark installation.
Reviewer guide
Validation
Windows, Node 26.13.0, standalone pnpm 11.9.0. Final reviewed head:
1748afb8767aa3ed86780da91f34f621dee368ec. The branch starts from refreshedmain; unrelated model-migration work is already merged and is not included here.pnpm run buildfromtsThe service suites are
topicWorkReuse,topicBuilds,topicSnapshot,topicCuration,topicGeneration,topicLookup,topicFrozenPreparation,topicCurationScheduler, andviewHostOptions, run withpnpm run jest-esm --runInBand --runTestsByPathagainst their correspondingdist\test\*.spec.jspaths.Replay tests and PR gates, from
ts:Tests cover exact receipt replay and cold resume, append/correction/forget behavior, audited-prior preservation, unknown compact IDs and bounded repair, sibling drainage, deadline enforcement before initialization/repairs/organization, metadata redaction/binding drift, and diagnostic organization without weakening native receipt checks. The captured-watermark regression is awaited so its lifetime cannot escape its parent test.
Live diagnostic results
One sequential, fixed-order control/candidate pair reused 2,853 raw topic nodes from 45 previously ingested chats; no new ingestion. Both arms used the same configured GPT-6 Luna model, unchanged provider-default reasoning, and independently matched 300-second HTTP request budgets under one unchanged absolute deadline. Earlier 60-second timeout failures were preserved, not mixed into this pair.
Candidate completed curation and actual organization in 40m03s. Curation latency was 48.8% lower, with 25.4% fewer calls and 30.8% fewer completion tokens; prompt tokens increased 0.7%.
Control's initial organization and bounded repair both returned hierarchy cycles, so the original control arm failed. A separate 1m03s, two-call live organization retry succeeded from its exact completed checkpoint, with zero repeated curation calls and no overlap with candidate inference. The original failed attempt remains a failure, not an uninterrupted successful pipeline.
Both final snapshots passed native snapshot/curation validation and accounted for all 2,853 inputs, with zero missing/duplicate/unknown assignments. Measurements were collected on the development snapshot before publication onto refreshed main; the final PR tree is rebuilt and offline-tested separately. No additional inference was authorized or started for PR preparation.
Risks, limitations and rollout