Skip to content

memory: use restricted Luna for topic reconciliation - #3288

Merged
Hillary Mutisya (hillary-mutisya) merged 3 commits into
mainfrom
dev/hillarym/topic-generative-decision-phase1
Nov 15, 2026
Merged

Hillary Mutisya (hillary-mutisya) merged 3 commits into
mainfrom
dev/hillarym/topic-generative-decision-phase1

Conversation

@hillary-mutisya

@hillary-mutisya Hillary Mutisya (hillary-mutisya) commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Use restricted-output generative judgments by default for prepared-topic reconciliation: admission, deferral and canonical group membership.
  • Keep the configured chat model (GPT-6 Luna by default), original evidence policy and generative authoring. Do not enable native Decision 1 or change preparation, summaries, editorial checks or organization.
  • Validate exact answer/option keys, numeric ranges and unit-sum distributions, then validate the joint partition and authored memberships. Invalid output takes an explicit, logged original-generative resolution path.
  • Add original-generative rollback, stage-specific policy/packet identities, schema-aware packing and frozen replay coverage.

Motivation and evidence

Separate bounded semantic decisions from descriptive authoring, making the decision contract explicit without migrating to the native Decision service.

A fresh matched diagnostic ran 12 source-disjoint tasks with four candidates each, three interfaces and three repeats: 108 model methods. All interfaces received identical caller-visible instructions, evidence and choices; the two Luna wire requests differed only in output schema.

Comparison Different selected choices
Full Luna A/A 23/144 (16.0%)
Restricted Luna A/A 14/144 (9.7%)
Full vs restricted Luna 28/144 (19.4%)
Restricted Luna vs native D1 85/144 (59.0%)

Restricted Luna passed 35/36 complete contracts. Excluding its one non-unit reply changes the full/restricted disagreement rate to 28/140 (20.0%). This supports the requested scoped rollout, not a correctness-equivalence claim: these are correlated comparisons of 48 unique candidates, the common diagnostic prompt differs from production framing, and agreement is not a quality label. Native D1's hidden templates/decoding were not observable, and only 28/36 native replies formed complete consistent partitions.

There is no end-to-end speed claim. An earlier small integrated pilot measured median full-construction times of 35.6 seconds for original generative and 51.8 seconds for the restricted path, including fallback (three runs per arm). The extra decision/authoring split and fallback work remain explicit.

Behavior and configuration

  • Default generativeDecisionStage: output-contract.
  • Rollback: TYPEAGENT_MEMORY_TOPIC_GENERATIVE_DECISION_STAGE=original, or topicCuration: { generativeDecisionStage: "original" }. The decomposed choice-only control remains available for diagnostics.
  • Restricted decisions return answers with typed choices, all offered probabilities and confidence. No probability repair, invented confidence, argmax threshold or source-specific rules are introduced.
  • Descriptive authoring must preserve the entire partition. Invalid contracts, inconsistent leaders, explicit synthesis or changed authored membership require original-generative resolution; authentication/provider/admission errors and cancellation propagate without authorizing another attempt.
  • The default policy gains :generative-partition-v1:output-contract. Original rollback preserves the old recipe. Changed policy/packet identities cannot resume original-policy work by relabeling it.
  • Context-aware packing considers the restricted schema, original fallback and a conservative authoring-payload bound.
  • Raw generation, custom topic adapters and opt-in native Decision execution retain their separate paths.

Reviewer guide

  1. Settings and policy identity, plus host configuration.
  2. Answer contract and decision, mapping, authoring and fallback boundary. The shared native validation error is exported solely for typed error handling.
  3. Curation integration and frozen preparation boundary: only reconciliation is adapted; installation remains atomic and uses existing validators.
  4. Contract/default/packing tests, existing topic generation/reuse tests, and frozen replay tests. Legacy response fixtures deliberately exercise explicit fallback and account for its extra calls.
  5. Service behavior and rollback and replay guide.

Validation

Tested commit 422666d74 on Windows, Node 26.13.0, standalone pnpm 11.9.0. All commands below run from ts, except the indicated package-local Jest commands.

  • pnpm run build: passed.
  • Memory-service Jest selection (topic.*|viewHostOptions)[.]spec[.]js$, with --runInBand: 318 tests in 17 suites.
  • Aiclient Jest selection decisionModels[.]spec[.]js$, with --runInBand: 96 tests.
  • pnpm run ci:node --test tools/memory-maintenance-qualification/topic-curation-options.test.mjs tools/memory-maintenance-qualification/topic-curation-replay.test.mjs: 28 tests.
  • pnpm run test:memory-quality: all 375 offline root-suite tests passed, including benchmark configuration identity.
  • pnpm run prettier:changed: passed.
  • pnpm run code-lint -- --ratchet --base origin/main: passed, zero violations across nine changed source files.
  • pnpm run code-complexity -- --ratchet --base origin/main --cyclomatic 25 --cognitive 30 --new-file-cyclomatic 25 --new-file-cognitive 30: passed, including both new source files.
  • pnpm run code-circular -- --ratchet --base origin/main --exceptions-file tools/scripts/code/circular-baseline-exception.json: passed, 210 cycles unchanged and 11 unchanged exceptions.
  • pnpm run code-debt -- --gate --base origin/main: passed, no focused/new skipped tests across 17 changed source/test files.

These are offline regression checks. No new live inference was run while preparing this PR; the measurements above are the completed diagnostic/pilot evidence, not tests of production readiness or latency on this final tree.

The first CI run caught one stale LongMemEval benchmark-default assertion on Linux Node 24 and 26. Commit 422666d74 updates that expected identity and checks that explicit original-generative rollback is recorded. The complete affected root suite, workspace build, topic/client tests, formatting and committed-source gates passed locally after the correction. Both Linux root suites now pass in CI on the follow-up head.

The later six build_ts failures all propagated from macOS Node 24 shard 4 being cancelled during dependency/runtime setup, before tests executed. After the workflow finished, a targeted retry succeeded: run 43767555530, attempt 2, on unchanged head 422666d74. The retried setup and test steps, sharded qualification, TypeScript CI and all six matrix checks passed. Current verified snapshot: 106 passing checks, three non-required skips, no failures or pending checks, and all required checks including smoke tests passing. No source or CI-policy changes were needed for this failure. Human approval remains required; monitoring has ended.

Risks and rollout

The decision/authoring split can add latency, and malformed numeric output can add fallback work. Numeric validity does not establish semantic quality. Monitor decision, authoring, fallback and installed-output outcomes separately. Retain the original rollback setting; do not treat native D1 as the replacement or silently weaken validation to improve completion rates.

Keep original generative curation as the default. Add opt-in decomposed and native-shaped GPT partition judgments with separate source-grounded authoring, strict answer validation and explicit resolution receipts. Preserve frozen preparation and reject admission failures without extra fallback attempts.

The bounded live pilot preserves ownership accounting but does not qualify default promotion: output probability arithmetic remains unreliable and total latency regresses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep preparation and descriptive authoring generative, validate bounded membership answers, and retain explicit original-generative resolution. Add a validated rollback setting and account for decision and authoring requests during token packing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hillary-mutisya

Copy link
Copy Markdown
Contributor Author

CI resolved on unchanged head 422666d74. The six build_ts failures all forwarded the shared qualification failure caused by macOS Node 24 shard 4 being cancelled during setup before tests executed. After the workflow finished, a targeted retry succeeded: run 43767555530, attempt 2. The macOS setup/test steps, qualification, TypeScript CI and all six matrix checks now pass. Verified 106 passing checks, three non-required skips, and no failed or pending checks; all required checks including smoke tests pass. No source or CI-policy changes were needed. Human approval remains required; nothing was merged.

@hillary-mutisya
Hillary Mutisya (hillary-mutisya) merged commit ef2da48 into main Nov 15, 2026
188 of 197 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant