Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions ts/examples/viewExamples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ after ingestion; they are not assumed to be memory revision IDs.
- `saved.json`: exact stored version, content and relationships.
- `rendered.txt`: product `viewContentToText` output, not an alternate schema.
- `request-evidence.json`: original indexed construction snapshots.
- `checkpoints.json`: the scheduled source batches, current stage and
not-attempted/running/accepted/failed disposition. A failed incremental build
retains earlier accepted revisions and leaves the later suffix unattempted.
- `receipt.json`: terminal build results, exact artifact hashes, mode,
injected behavior, authorization identity and publication/index state.
- `failure.json`: explicit component, stage and reason after a store was opened.
Expand Down Expand Up @@ -112,6 +115,53 @@ or deliverables. Use sufficiently descriptive approved sources for a useful
project-understanding evaluation. Live execution alone does not make the original
development fixtures reviewed quality gold.

### Opt-in admitted-request capture

Add `--capture-requests` to an already authorized live command to retain exact
post-redaction request bodies after admission and adaptive output-capacity
changes. Offline mode refuses this flag. Capture is scoped to the run; normal
execution does not write these files:

- `calls/<attempt-id>/request.json`: the exact JSON body sent for one physical
attempt, not concatenated calls, inspection output or a reconstructed prompt.
- `request-calls.json`: contract `memory-request-capture-v1`, case/checkpoint,
operation, SHA-256 body hash, reservations, dispatch outcome and measured usage
when the completion integration supplies it. Unknown usage is `null`, not zero.
- `calls/<attempt-id>/result.json`: the model completion result when forwarded
by informational/topic integration, with its separate manifest hash. These
are untrusted candidates/checker results, **not** accepted revisions or raw
HTTP response envelopes; they survive later validation rejection.

Retries receive distinct attempt IDs. Construction, contract repair and audit
operations are distinct; auditor-only evidence cannot establish construction
selection recall. Manifest/body persistence failures stop dispatch explicitly.
Source text remains sensitive even after normal secret filtering: keep capture
files private and local, and never commit or publish them.

This capture does not authorize inference or turn a runner receipt into a quality
baseline. Arm/generation/provider identities still belong to the separately frozen
campaign run. Ingestion, embeddings, failed responses and integrations without
usage forwarding can have unknown usage; reservations are estimates, not measured
tokens or billed cost. Capture supplies neither independent semantic judgments nor
automatic cost estimates.

Captured runs also retain `evidence.json` with the actual original indexed block
and service-resolution receipt for each selected/cited locator. Each accepted
wiki/project checkpoint exports its exact saved/rendered revision beneath
`revisions/`; the cumulative mapping and construction snapshots survive a later
failure. `getSourceContent({indexedLocator, includeIndexedBlock: true, ...})`
provides this opt-in receipt under the same source authorization and revision
checks; ordinary source expansion is unchanged.

For approved isolated multi-arm execution, use the existing
[campaign tooling](../../tools/memory-maintenance-qualification/quality-view-campaign-README.md).
The CLI supports `--request-budget <file>` with `--capture-requests` and
`--expected-runtime <frozen-arm.json>` to reject runtime drift before inference.
`--runtime-identity` alone reads configured settings and emits only non-secret
names/hashes without inference; it is separate from credential-free help/list.
Campaign budgets bound attempts, conservative token reservations and an
owner-approved maximum per-call cost reservation, not measured billing.

## Dataset and evaluation tooling

Separate offline preparation and campaign tools live in
Expand Down
108 changes: 108 additions & 0 deletions ts/examples/viewExamples/src/campaignBudget.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

import { RequestBudget } from "@typeagent/aiclient";

export function configuredModelIdentity(settings: {
provider: string;
modelName?: string | undefined;
endpoint: string;
}): string | null {
if (settings.modelName?.trim()) return settings.modelName;
if (settings.provider !== "azure") return null;
const deployment = new URL(settings.endpoint).pathname.match(
/\/openai\/deployments\/([^/]+)(?:\/|$)/,
)?.[1];
return deployment ? decodeURIComponent(deployment) : null;
}

export function assertCampaignRuntime(
actual: {
configurationIdentity: string;
configurationHash: string;
providerSettingsHash: string;
nodeVersion: string;
providers: (string | null)[];
models: (string | null)[];
},
expected: unknown,
) {
if (!expected || typeof expected !== "object")
throw new Error("Frozen arm runtime required");
const frozen = expected as Record<string, unknown>;
if (
[
"configurationIdentity",
"configurationHash",
"providerSettingsHash",
"nodeVersion",
].some((key) => frozen[key] === undefined)
)
throw new Error("Incomplete frozen runtime");
if (
actual.configurationIdentity !== frozen.configurationIdentity ||
actual.configurationHash !== frozen.configurationHash ||
actual.providerSettingsHash !== frozen.providerSettingsHash ||
actual.nodeVersion !== frozen.nodeVersion ||
!actual.providers.length ||
!actual.models.length ||
actual.providers.some(
(provider) => provider === null || provider !== frozen.provider,
) ||
actual.models.some((model) => model === null || model !== frozen.model)
)
throw new Error(
"Configured runtime differs from frozen arm; no dispatch authorized",
);
}

export function parseCampaignBudget(value: unknown) {
if (!value || typeof value !== "object")
throw new Error("Campaign request budget required");
const input = value as Record<string, unknown>;
const integer = (key: string) => {
const number = input[key];
if (
typeof number !== "number" ||
!Number.isSafeInteger(number) ||
number < 1
)
throw new Error(
`Campaign budget.${key}: positive integer required`,
);
return number;
};
if (input.tokenizer !== "utf8-bytes-v1")
throw new Error(
"Campaign requires the named conservative utf8-bytes-v1 estimator",
);
const calls = integer("calls");
const reservedTokens = integer("reservedTokens");
const cost = input.cost;
const maxCostPerCall = input.maxCostPerCall;
if (
typeof cost !== "number" ||
!Number.isFinite(cost) ||
cost <= 0 ||
typeof maxCostPerCall !== "number" ||
!Number.isFinite(maxCostPerCall) ||
maxCostPerCall <= 0
)
throw new Error(
"Positive cost reservation and approved per-call upper bound required",
);
return {
budget: new RequestBudget({
inputTokens: integer("inputTokens"),
outputTokens: integer("outputTokens"),
safetyTokens: integer("safetyTokens"),
chatAttempts: calls,
embeddingAttempts: calls,
reservedTokens,
elapsedMs: integer("elapsedMs"),
tokenizer: input.tokenizer,
countTokens: (serialized) => Buffer.byteLength(serialized, "utf8"),
}),
captureLimits: { calls, reservedTokens, cost, maxCostPerCall },
};
}
150 changes: 133 additions & 17 deletions ts/examples/viewExamples/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,61 @@ import { readFile } from "node:fs/promises";
import { parseArgs } from "node:util";
import { pathToFileURL } from "node:url";
import { createConfiguredViewSynthesisAdapter } from "@typeagent/memory-service";
import { initRuntimeConfigFromProcessEnv, openai } from "@typeagent/aiclient";
import {
initRuntimeConfigFromProcessEnv,
openai,
runWithRequestBudget,
} from "@typeagent/aiclient";
import { loadConfigSync } from "@typeagent/config";
import { originalFixtures } from "./fixtures.js";
import { parseConstructionPacket } from "./packets.js";
import { digest, parseConstructionPacket } from "./packets.js";
import { runInformational } from "./runner.js";
import {
assertCampaignRuntime,
configuredModelIdentity,
parseCampaignBudget,
} from "./campaignBudget.js";

export const usage = `wiki-views --mode offline|live --case <id> --root <new-example-local-run> [--input construction.json] [--authorization approval.json]
export const usage = `wiki-views --mode offline|live --case <id> --root <new-example-local-run> [--input construction.json] [--authorization approval.json] [--capture-requests]
wiki-views --list
wiki-views --runtime-identity
Offline uses deterministic injection, not model-quality evidence.
Live requires a separate approved configuration and never falls back to offline.`;

function configuredRuntime() {
loadConfigSync();
initRuntimeConfigFromProcessEnv();
if (!openai.hasChatModelEndpoint())
throw new Error("Live chat model configuration is missing");
const adapter = createConfiguredViewSynthesisAdapter();
const members = openai.getChatModelPool().members.map(({ settings }) => ({
provider: settings.provider,
modelName: settings.modelName,
endpoint: settings.endpoint,
wireApi: settings.wireApi,
requestConstraints: settings.requestConstraints,
}));
return {
adapter,
runtime: {
configurationHash: digest(
JSON.stringify({
synthesisIdentity: adapter.identity,
informationalOutputTokens: 4000,
maintenanceMode: "manual",
discoveryPolicy: "off",
publication: false,
}),
),
providerSettingsHash: digest(JSON.stringify(members)),
providers: [
...new Set(members.map((member) => member.provider ?? null)),
],
models: [...new Set(members.map(configuredModelIdentity))],
},
};
}

export async function main(args = process.argv.slice(2)) {
const { values } = parseArgs({
args,
Expand All @@ -27,9 +71,23 @@ export async function main(args = process.argv.slice(2)) {
root: { type: "string" },
input: { type: "string" },
authorization: { type: "string" },
"capture-requests": { type: "boolean" },
"request-budget": { type: "string" },
"runtime-identity": { type: "boolean" },
"expected-runtime": { type: "string" },
},
});
if (values.help || !args.length) return usage;
if (values["runtime-identity"]) {
if (args.length !== 1)
throw new Error("--runtime-identity must be used alone");
const { runtime, adapter } = configuredRuntime();
return {
...runtime,
nodeVersion: process.version,
configurationIdentity: adapter.identity,
};
}
const packet: unknown = values.input
? JSON.parse(await readFile(values.input, "utf8"))
: originalFixtures();
Expand All @@ -44,6 +102,15 @@ export async function main(args = process.argv.slice(2)) {
(values.mode !== "offline" && values.mode !== "live")
)
throw new Error("--case, --root and explicit --mode required");
if (
values.mode === "offline" &&
(values["capture-requests"] ||
values["request-budget"] ||
values["expected-runtime"])
)
throw new Error(
"Admitted request capture requires authorized live mode",
);
if (values.mode === "offline")
return (
await runInformational({
Expand All @@ -53,10 +120,30 @@ export async function main(args = process.argv.slice(2)) {
mode: "offline",
})
).receipt;
if (!values.authorization)
return runLive({
packet,
caseId: values.case,
root: values.root,
authorization: values.authorization,
captureRequests: values["capture-requests"],
requestBudget: values["request-budget"],
expectedRuntime: values["expected-runtime"],
});
}

async function runLive(options: {
packet: unknown;
caseId: string;
root: string;
authorization: string | undefined;
captureRequests: boolean | undefined;
requestBudget: string | undefined;
expectedRuntime: string | undefined;
}) {
if (!options.authorization)
throw new Error("Live mode requires --authorization");
const approval: unknown = JSON.parse(
await readFile(values.authorization, "utf8"),
await readFile(options.authorization, "utf8"),
);
if (
!approval ||
Expand All @@ -70,21 +157,50 @@ export async function main(args = process.argv.slice(2)) {
throw new Error(
"Live approval requires approved: true and a non-secret identity",
);
loadConfigSync();
initRuntimeConfigFromProcessEnv();
if (!openai.hasChatModelEndpoint())
throw new Error("Live chat model configuration is missing");
return (
await runInformational({
packet,
caseId: values.case,
root: values.root,
const configured = configuredRuntime();
if (options.expectedRuntime) {
if (!options.requestBudget || !options.captureRequests)
throw new Error(
"Frozen campaign execution requires request capture and budgets",
);
assertCampaignRuntime(
{
...configured.runtime,
nodeVersion: process.version,
configurationIdentity: configured.adapter.identity,
},
JSON.parse(await readFile(options.expectedRuntime, "utf8")),
);
}
const budget = options.requestBudget
? parseCampaignBudget(
JSON.parse(await readFile(options.requestBudget, "utf8")),
)
: undefined;
if (budget && !options.captureRequests)
throw new Error("Campaign budgets require --capture-requests");
const authorizationIdentity = approval.identity;
const execute = () =>
runInformational({
packet: options.packet,
caseId: options.caseId,
root: options.root,
mode: "live",
liveAuthorization: { approved: true, identity: approval.identity },
captureRequests: options.captureRequests ?? false,
...(budget ? { captureLimits: budget.captureLimits } : {}),
runtimeIdentity: configured.runtime,
liveAuthorization: {
approved: true,
identity: authorizationIdentity,
},
serviceOptions: {
viewSynthesisAdapter: createConfiguredViewSynthesisAdapter(),
viewSynthesisAdapter: configured.adapter,
},
})
});
return (
await (budget
? runWithRequestBudget(budget.budget, execute)
: execute())
).receipt;
}

Expand Down
Loading
Loading