Skip to content

memory: use choice-only Luna verdicts for bounded Wiki decisions - #3287

Merged
Hillary Mutisya (hillary-mutisya) merged 1 commit into
mainfrom
dev/hillarym/wiki-luna-choice-only
Nov 15, 2026
Merged

Hillary Mutisya (hillary-mutisya) merged 1 commit into
mainfrom
dev/hillarym/wiki-luna-choice-only

Conversation

@hillary-mutisya

Copy link
Copy Markdown
Contributor

Summary

  • Default bounded Wiki decisions to GPT_6_LUNA with strict choice-only answers: { answers: { questionId: { type: "choice", choice: "<offered value>" } } }. Reject probabilities, confidence, extra fields, missing answers and unoffered choices.
  • Preserve existing evidence preparation, exact prose-witness binding, host validation and prose generation. Independently check utility, admission/canonical selection, contribution coverage and consolidation relation; fail explicitly if the restricted verdict disagrees with the validated preparation rather than inventing matching metadata.
  • Route Wiki editorial/consolidation verification through bounded safety/category choices. Completeness can update status only, never generated prose or citations.
  • Include the opt-in three-arm replay/fresh-data tooling and deterministic regression coverage.

Motivation

These boundaries need categorical decisions, not generated probability distributions or confidence claims. Restricting the response contract makes the accepted output explicit while retaining the richer evidence required by the existing host guards.

A preceding component diagnostic used six complete repository documents, 36 bounded packets and three repeats across native, full-shaped Luna and choice-only Luna arms. All 324 scored outputs were valid, but the Luna schemas disagreed on 13/126 question pairs and native/full on 32/126. There were no independent correctness labels or repeat-stable cross-schema differences. This motivates an explicit contract, not an accuracy, equivalence or performance claim. The production integration in this PR is offline-tested; that diagnostic was not an end-to-end test of these new gates.

Behavior and configuration

  • Enabled by default for the configured Wiki synthesis adapter; this is not an opt-in rollout.
  • Requires an existing Azure/OpenAI GPT_6_LUNA endpoint. Restricted calls use temperature one and explicitly opt into the SDK's strict-schema path. No new credentials, configuration files or dependencies are added.
  • Evidence preparation remains on the existing configured model. Its proposed dispositions and metadata cannot be accepted unless the independent restricted verdict agrees. Utility receives original evidence without page context. Uncertainty/disagreement is a visible operation failure, not a fallback or automatic outcome retry.
  • Existing evidence-contract repairs remain bounded and unchanged; they cannot repair/resample a rejected restricted verdict into acceptance. Editorial safe/category consistency is still checked by the existing parser.
  • Prose writers/repairs, project-summary decisions, non-Wiki contracts and topic curation keep their existing models/contracts. Custom host adapters are not replaced.
  • Additional calls use the existing maintenance scheduler, cancellation and provider request budgets. The configured adapter identity now covers both model bindings and invalidates the previous cached assessment policy.

Reviewer guide

  1. Choice-only schema and parser: exact question IDs, finite offered choices and closed objects; no score synthesis.
  2. Configured synthesis adapter: Luna binding and the six bounded roles, existing preparation/repair paths, explicit disagreement handling and cache identity.
  3. Integration/contract tests: actual SDK wire inspection, independent conflicting verdicts, unchanged prose, cancellation and failure propagation. Existing admission/consolidation fixtures now script matching restricted answers; these are synthetic tests, not quality evidence.
  4. Operator behavior and replay runbook: current defaults, limitations and separately approved live execution.

The replay runner has its own fixed 327-call schedule and immutable evidence/protocol checks. The current fresh-data driver also counts the newly added six completeness calls: expected total 375 within one 420-call/90-minute ceiling. It does not silently reconstruct a missing historical dataset or qualify host consequences.

Validation

Validated on Windows, Node 26 and standalone pnpm, at e5213de7d920bb08008f6d2e90343d6914f1ec3b:

Check Result
pnpm run build PASS, full workspace
Affected memory-service suites below PASS, 10 suites / 339 tests, including 20 new choice-only tests
pnpm run test:memory-quality PASS, 392 tests, including 17 schema/fresh-driver tests
pnpm run prettier:changed PASS
Lint ratchet against origin/main PASS, zero new violations
Complexity ratchet, cyclomatic 25 / cognitive 30, including new files PASS
Circular-dependency ratchet with existing baseline exceptions PASS, no new cycles
Debt gate PASS, no focused/newly skipped tests
git diff --check PASS

From ts, the affected suites are reproducible with:

pnpm --filter @typeagent/memory-service run jest-esm --runInBand `
  '--testPathPattern=(wikiDecision|wikiAdmission|wikiConsolidation|viewSynthesisStructural|schemaRequestAdmission|informationalProjectPrompt|informationalWiki|knowledgeDiscovery|schemaAnswerRepair|sourceSynopsis)[.]spec[.]js$'

The first local circular scan raced a test-owned temporary Git lock directory being deleted. Re-running after tests completed passed without changing the scanner or its baseline. No live inference was run to validate the production integration.

Risks and rollback

This adds model calls and can reject assessments previously accepted by preparation alone. Cost, rejection rate and end-to-end latency/quality have not been qualified; no speedup is claimed. Provider output admission allowance is not a hard wire generation cap. Source-backed correctness adjudication and broader live host qualification remain follow-ups.

Rollback is a revert of this policy change. The PR does not change global model defaults, activate publication authority, weaken evidence guards, or enable automatic merging.

Enable strict bounded Wiki verdicts without probabilities or confidence. Preserve evidence preparation, exact witness guards and prose generation, and reject verdict/evidence disagreement explicitly. Add regression coverage and a bounded three-arm replay harness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hillary-mutisya
Hillary Mutisya (hillary-mutisya) merged commit 6e8257c into main Nov 15, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant