Skip to content

memory: reduce repeated topic curation with frozen preparation - #3282

Merged
Hillary Mutisya (hillary-mutisya) merged 2 commits into
mainfrom
dev/hillarym/memory-topic-curation
Nov 11, 2026
Merged

Hillary Mutisya (hillary-mutisya) merged 2 commits into
mainfrom
dev/hillarym/memory-topic-curation

Conversation

@hillary-mutisya

Copy link
Copy Markdown
Contributor

Summary

  • Add opt-in compact wire IDs, prior-concept delta responses, and two-worker frozen topic preparation followed by serialized, validated reconciliation.
  • Preserve exact evidence/membership checks, ordered window receipts, checkpoint recovery, cancellation/draining, and native publication/freshness guards.
  • Harden retained-input topic replay with maintained history reads, pinned model metadata, absolute deadlines, validated partial outcomes, and the corrected diagnostic organization handoff.

Motivation

Serial topic curation repeatedly sends and rewrites already-admitted concepts. Speculative preparation can also become invalid when earlier decisions change its prior-concept frontier. Prepare independent raw packets against frozen inputs instead, then reconcile their proposals against current admitted concepts before installing a complete window.

Behavior and configuration

All three new treatments are off by default:

Environment variable Behavior
TYPEAGENT_MEMORY_TOPIC_PRIOR_DELTAS Omit unchanged prior concepts from responses; the host retains their exact objects, including audited summaries.
TYPEAGENT_MEMORY_TOPIC_COMPACT_IDS Remap packet-local wire IDs and schema enums reversibly; evidence, prose, membership, and durable IDs stay unchanged.
TYPEAGENT_MEMORY_TOPIC_FROZEN_PREPARATION Prepare two raw-only packets concurrently, then serialize reconciliation and complete-window installation. Requires explicit concurrency 2.

These are also programmatic topicCuration settings and replay CLI switches. Effective treatment settings participate in policy/replay identity. Watermark continuation accepts explicit frozen two-worker execution, not ordinary speculative concurrency or the native Decision engine. Source correction/removal, forgetting, eligibility, and current-publication checks remain fail-closed.

Automatic maintenance may recapture a changed topic head once during admission; repeated churn still fails stale. Explicit callers retain strict expected-head semantics.

The replay organization handoff drops captured native packet-receipt assertions from its cloned diagnostic input. It does not disable production receipt validation or certify native watermark installation.

Reviewer guide

  1. Curation orchestration, frozen preparation, and wire remapping.
  2. Host integration in fileMemoryService.ts, viewHostOptions.ts, and topicGeneration.ts.
  3. Replay driver, replay documentation, and regression tests.

Validation

Windows, Node 26.13.0, standalone pnpm 11.9.0. Final reviewed head: 1748afb8767aa3ed86780da91f34f621dee368ec. The branch starts from refreshed main; unrelated model-migration work is already merged and is not included here.

Check Result
Full pnpm run build from ts Pass
Nine focused memory-service Jest suites 169/169 pass
Three topic replay/options Node test suites 43/43 pass; no skips/cancellations
Pinned changed-file formatting Pass
Committed-HEAD lint, complexity, circular and test-debt gates All pass: zero added lint/complexity violations; cycles 210 to 210; no focused/newly skipped tests

The service suites are topicWorkReuse, topicBuilds, topicSnapshot, topicCuration, topicGeneration, topicLookup, topicFrozenPreparation, topicCurationScheduler, and viewHostOptions, run with pnpm run jest-esm --runInBand --runTestsByPath against their corresponding dist\test\*.spec.js paths.

Replay tests and PR gates, from ts:

node --test .\tools\memory-maintenance-qualification\topic-curation-replay.test.mjs .\tools\memory-maintenance-qualification\topic-decision-replay.test.mjs .\tools\memory-maintenance-qualification\topic-curation-options.test.mjs
pnpm run prettier:changed
pnpm run code-lint -- --ratchet --base origin/main
pnpm run code-complexity -- --ratchet --base origin/main --cyclomatic 25 --cognitive 30 --new-file-cyclomatic 25 --new-file-cognitive 30
pnpm run code-circular -- --ratchet --base origin/main --exceptions-file tools\scripts\code\circular-baseline-exception.json
pnpm run code-debt -- --gate --base origin/main

Tests cover exact receipt replay and cold resume, append/correction/forget behavior, audited-prior preservation, unknown compact IDs and bounded repair, sibling drainage, deadline enforcement before initialization/repairs/organization, metadata redaction/binding drift, and diagnostic organization without weakening native receipt checks. The captured-watermark regression is awaited so its lifetime cannot escape its parent test.

Live diagnostic results

One sequential, fixed-order control/candidate pair reused 2,853 raw topic nodes from 45 previously ingested chats; no new ingestion. Both arms used the same configured GPT-6 Luna model, unchanged provider-default reasoning, and independently matched 300-second HTTP request budgets under one unchanged absolute deadline. Earlier 60-second timeout failures were preserved, not mixed into this pair.

Metric Control Candidate
Curation including setup 76m32s 39m12s
Curation model-method calls 134 100
Prompt tokens 1,553,127 1,564,108
Completion tokens 371,313 257,085
Final concepts 269 201
Admitted / explicitly deferred raw nodes 2,559 / 294 2,554 / 299

Candidate completed curation and actual organization in 40m03s. Curation latency was 48.8% lower, with 25.4% fewer calls and 30.8% fewer completion tokens; prompt tokens increased 0.7%.

Control's initial organization and bounded repair both returned hierarchy cycles, so the original control arm failed. A separate 1m03s, two-call live organization retry succeeded from its exact completed checkpoint, with zero repeated curation calls and no overlap with candidate inference. The original failed attempt remains a failure, not an uninterrupted successful pipeline.

Both final snapshots passed native snapshot/curation validation and accounted for all 2,853 inputs, with zero missing/duplicate/unknown assignments. Measurements were collected on the development snapshot before publication onto refreshed main; the final PR tree is rebuilt and offline-tested separately. No additional inference was authorized or started for PR preparation.

Risks, limitations and rollout

  • This is a combined-treatment, single-pair observation, not individual-factor causal attribution, a native required-readiness result, or semantic/answer-quality qualification.
  • Candidate produced 68 fewer concepts and deferred five additional raw nodes. Structural coverage cannot distinguish useful consolidation from over-merging; semantic review remains a release gate.
  • HTTP transport settings are not covered by the model-binding hash and must be matched independently. Production timeout defaults are unchanged.
  • No experimental defaults are promoted. Roll back by unsetting the three treatment variables; changed policy/progress identities fail closed rather than mixing incompatible checkpoints.
  • Corpus payloads, private archives, credentials, and machine-local links are intentionally excluded.

Add opt-in prior deltas, compact wire IDs and two-worker frozen preparation with serialized, validated reconciliation and ordered watermark receipts.

Harden retained-input replay with model bindings, absolute deadlines, validated partial progress and the corrected diagnostic organization handoff. Preserve default behavior and native receipt validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep deadline cancellation artifacts and persistence failures unchanged while bringing the replay lifecycle within the committed-branch complexity budget.

Validation: full workspace build and all 43 replay/options tests pass; the 169 service tests passed on the unchanged service implementation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hillary-mutisya
Hillary Mutisya (hillary-mutisya) merged commit 3317b2a into main Nov 11, 2026
109 checks passed
Hillary Mutisya (hillary-mutisya) added a commit that referenced this pull request Nov 13, 2026
## Summary

- Make frozen two-packet preparation the only generative topic-curation
schedule, followed by ordered reconciliation and complete-window
commits.
- Remove the legacy serial/speculative scheduler and its configuration
selectors.
- Align production, replay, watermark and benchmark identities with the
new defaults while preserving native Decision execution.

## Motivation

#3282 added frozen preparation, but ordinary production configuration
still selected the old serial path. This promotes the frozen
implementation instead of retaining competing schedules that can
repeatedly invalidate speculative proposals.

## Behavior and migration

- Prior deltas and compact wire IDs now default to `true`. Input-token
targets and reserved prior fractions remain optional; this does **not**
make the historical 24,000-token / 0.25-fraction treatment the default.
- Remove `TYPEAGENT_MEMORY_TOPIC_CONCURRENCY` and
`TYPEAGENT_MEMORY_TOPIC_FROZEN_PREPARATION`, including their former
candidate values. Their programmatic counterparts, `concurrency` and
`frozenPreparation`, also fail explicitly.
- Remove replay flags `--concurrency`, `--frozen-preparation` and
`--no-frozen-preparation`.
- Remove the `topic-parallel` experiment. `indexing-parallel` now
selects only synopsis and embedding treatments; indexing ablation no
longer offers the obsolete `T1` arm.
- The new `curated-lazy-v10:<hash>` recipe prevents older-policy
progress/receipts from qualifying as current work. Recorded replay
requires a matching recording identity; historical progress is rejected
rather than relabeled.
- Reuse and private watermarks remain opt-in. Failed/cancelled
preparation cannot checkpoint a partial window; exact validated prefixes
can still be reused.
- Serial reconciliation and installation are intentional. The shared
single-pass evidence primitive remains necessary inside frozen
preparation/reconciliation and native uncertainty resolution; it is not
an alternate host scheduling mode.
- Native Decision concurrency and its existing resolution behavior
remain unchanged. The explicit raw diagnostic path also remains
available.

## Reviewer guide

1. [Settings and policy
identity](ts/packages/memory/service/src/topicCurationSettings.ts), then
[generative
orchestration](ts/packages/memory/service/src/topicCuration.ts).
2. [Frozen
windows](ts/packages/memory/service/src/topicFrozenPreparation.ts) and
[native uncertainty
resolution](ts/packages/memory/service/src/topicDecisionCuration.ts).
Evidence validation continues to reuse the snapshot-local index.
3. [Service
wiring](ts/packages/memory/service/src/fileMemoryService.ts), [host
configuration](ts/packages/memory/service/src/viewHostOptions.ts), and
[captured-work reuse](ts/packages/memory/service/src/topicCapture.ts).
4. [Replay/migration
guide](ts/tools/memory-maintenance-qualification/topic-curation-README.md),
replay identity/capture tests, and service tests for window draining,
cancellation, resumptions, append/correction/forget churn and exact
output parity.

## Validation

Initial promotion validated at `fc54c3cee` on Windows against
`origin/main` at `3317b2a4a`.

| Check | Result |
| --- | --- |
| `pnpm run build` | Passed, 104 tasks |
| All topic service suites plus host options, maintenance priority and
performance experiments | 321 passed, 17 suites |
| `pnpm run test:memory-quality` | 375 passed, no skips |
| `pnpm run ci:node --test
tools/memory-maintenance-qualification/topic-work-benchmark.test.mjs` |
3 passed; exact frozen call counts, prefix reuse and output parity |
| Repository-pinned Prettier and `git diff --check` | Passed |
| Committed-HEAD lint, complexity, circular-dependency and test-debt
gates against `origin/main` | All four passed; no new violations, cycles
or skips |

Service test selection, from `ts`, in PowerShell:

```powershell
Set-Location packages\memory\service
$tests = @(Get-ChildItem dist\test\topic*.spec.js | ForEach-Object { $_.FullName })
$tests += @('viewHostOptions', 'maintenanceBuildPriority', 'memoryPerformanceExperiments') |
    ForEach-Object { (Resolve-Path "dist\test\$_.spec.js").Path }
pnpm run jest-esm --runTestsByPath @tests
```

Additional legacy offline-harness suite: **9 passed, 1 pre-existing
failure**. Its worker-bootstrap test fails before the intended
missing-profile check because unchanged trace instrumentation expects
the absent `runner.verifyContributionRefresh` hook. Running the original
`origin/main` harness assertions reproduces the same failure (8 passed,
1 failed); this is not a clean-baseline full-build claim. The newly
added compiled-settings provenance assertion passes. This unrelated
harness issue is not repaired here.

## CI repair at `c8b1f8aa3`

The Windows Node 24 remainder shard failed in browser
`viewLifecycle.test.mjs`: the real child view server exceeded its
existing 10-second readiness deadline. It eagerly loaded the entire
memory runtime twice through independent import paths: experiment
configuration in the gateway and paging constants in the memory client.

The follow-up exposes lightweight `performance-experiments` and
`view-detail` service entry points and uses them at those boundaries.
Existing root exports and configuration behavior are preserved. A real
ESM loader-hook regression rejects accidental imports of the full
memory, indexing or model runtimes during gateway startup. It failed
before the repair and passes afterward.

The production readiness deadline and lifecycle-test timeout are
**unchanged**; no workflow, retry budget or test coverage was weakened.

Local validation of the repair on Windows, Node 26.13.0:

- `pnpm run build`: passed across the full workspace.
- Browser `pnpm run test:local`: **1,071 passed**, including all 130
gateway tests and all 516 view-client tests.
- Memory client `pnpm run test:local`: **81 passed**, including
pagination/protocol coverage.
- Memory service host/configuration suites: **33 passed**.
- Repository-pinned formatting, whitespace check and all four
committed-HEAD ratchets: passed.

The new CI run must verify the originally failing Windows Node 24 shard;
local validation is not a claim that the remote run has completed.

## Risks and follow-up

- No new live inference or latency/semantic-quality qualification was
run for this promotion. Synthetic fixtures establish scheduling,
accounting and reuse invariants, not semantic equivalence or production
speedup.
- Frozen preparation introduces a preparation/reconciliation boundary
and can require extra calls for small or all-deferred inputs. Historical
candidate timings are not measurements of these default packet budgets.
- Existing deployments with removed selectors must clear them.
Older-policy curation may need rebuilding.
- Rollback is a code revert, not a legacy scheduling flag. No automatic
merge is requested.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant