diff --git a/README.md b/README.md index 5a14b29..259abec 100644 --- a/README.md +++ b/README.md @@ -106,17 +106,24 @@ In Claude Code, you can also run: /behavior-diff ``` -Behavior Diff finds the changed instruction file and uses your request as the -comparison task. Once the task is known, and any shared trial data is verified, -it runs the comparison and opens the report. +Behavior Diff finds the changed instruction file and works out what the change +is meant to do and what it must preserve. From that intent, the agent designs a +neutral task and, when needed, coherent ordinary and edge-case test data, plus +private criteria for each case. It reuses suitable project data when it can and +asks you only when a needed requirement cannot be found another way. It checks +the environment, freezes the task and data, runs the comparison, and opens the +report. It then explains, separately from the automatic report, which cases +were reached and whether the evidence supports the intent. + +This coverage is bounded to the designed cases. It is not an exhaustive test, +and it does not promise to find unknown side effects. ### Shared trial data Some tasks need files that your project does not contain, such as sample inputs. Git-based trials start from your last commit, so untracked or -uncommitted files in your checkout never reach them. Behavior Diff does not -generate such data. When you supply or approve the files, the skill can deliver -them to every trial: +uncommitted files in your checkout never reach them. The skill designs or +selects these files; a helper only delivers them to every trial: 1. A private helper, `prepare_shared.py`, copies your project base, the reviewed data directory, and the selected Before and After instructions into a new diff --git a/docs/architecture.md b/docs/architecture.md index 38f8792..e7d222b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,20 +18,18 @@ scripts handle execution, evidence, and reporting. +----------------------------+----------------------------+ v +---------------------------------------------------------+ -| Skill orchestration | -| Select edit/scenario; freeze purpose + source revisions | +| Skill preparation (LLM; reference recipe) | +| Change intent + preservation -> capabilities/premises | +| -> neutral task + ordinary/bounded edge data + private | +| case criteria; reuse sufficient project data | +| Preflight host environment; freeze task, data, purpose, | +| source revisions, criteria, and execution settings | +| Shared files or exclusions: prepare_shared.py stages | +| and verifies; otherwise runner uses user's project | +----------------------------+----------------------------+ v +---------------------------------------------------------+ -| Conditional shared-data delivery | -| Only when the task needs files the project base lacks: | -| prepare_shared.py freezes project, Before, and home; | -| verify passes immediately before launch | -| Otherwise the runner starts from the user's project | -+----------------------------+----------------------------+ - v -+---------------------------------------------------------+ -| Trial execution | +| Runner: default 3+3 trials | | behavior-diff.sh builds independent Before/After copies | | run-trial.sh runs the same task/model in each copy | +----------------------------+----------------------------+ @@ -51,7 +49,15 @@ scripts handle execution, evidence, and reporting. +---------------------------------------------------------+ | Presentation | | render.py -> HTML, Markdown, and structured JSON | -| Skill explains evidence; user reviews the change | +| Automatic report from saved evidence | ++----------------------------+----------------------------+ + v ++---------------------------------------------------------+ +| Skill result explanation (separate from the report) | +| Raw actions/returns/artifacts per private case: | +| reached / missed / unknown; intent supported / | +| contradicted / insufficient evidence; run health apart; | +| report agreement or gap disclosed; user decides | +---------------------------------------------------------+ ``` @@ -113,17 +119,31 @@ requires explicit user acceptance. Hook failures do not block the host session. ## 2. The skill owns the experiment [`SKILL.md`](../plugin/skills/behavior-diff/SKILL.md) guides the host agent to -select the changed file and prepare a scenario at the relevant decision point. -The task must not reveal the expected behavior. The instruction change must -remain the only source of that guidance. - -The skill selects the trial host and model, calls the runner, and explains the -result. It owns scenario quality and interpretation, not filesystem copying or -report formatting. When the task needs common files that the project base -lacks, the skill reviews files that the user supplied or approved and delivers -them through a frozen stage (see -[Shared-data delivery](#shared-data-delivery)). The skill does not generate the -scenario or the data. +select the changed file and prepare the comparison from the change's intent. The +agent itself designs the scenario and any test data; the +[skill reference](../plugin/skills/behavior-diff/references/scenario-and-test-data.md) +holds the detailed recipe. In short, the agent derives the intended change and +the behavior it must preserve, the capabilities and premises that behavior +needs, a neutral task that starts before the tested work, coherent ordinary and +bounded edge-case data, and private per-case criteria. It reuses sufficient +project data instead of generating more. It asks the user only when a material +requirement cannot be reached otherwise. The task must not reveal the expected +behavior. The instruction change must remain the only source of that guidance. + +Before launch, the skill preflights the real host environment and freezes the +task, data, criteria, purpose, source revisions, and execution settings. It then +runs the existing runner with the default trial counts; more edge cases add +coverage within those trials rather than more runs. When common files must be +added or private paths excluded, the skill delivers reviewed data through a +frozen stage (see [Shared-data delivery](#shared-data-delivery)). It owns scenario +quality and interpretation, not filesystem copying or report formatting. + +After the run, the skill explains the result separately from the automatic +report: per private case it reads raw actions, returns, and artifacts, then +states whether each case was reached, missed, or unknown, and whether the intent +is supported, contradicted, or lacks evidence. Run health stays separate from +requirement failures. The coverage is bounded to the designed cases; it does +not prove the absence of unknown side effects. Before execution, the skill records distinct concise goals and their provenance (`session`, `commit`, or `diff`; `explicit` or `inferred`) in a reviewed purpose @@ -286,8 +306,10 @@ shared-data stage. The private helper [`prepare_shared.py`](../plugin/skills/behavior-diff/scripts/prepare_shared.py) delivers a reviewed data directory to every trial without changing the user's -checkout or adding runner options. It does not generate data, launch models, -or infer provenance. +checkout or adding runner options. It only delivers and verifies: the skill +designs the data, and the helper does not generate data, launch models, or infer +provenance. Its freeze covers the stage, not the private task, purpose, source +metadata, or tooling; the skill checks those separately before launch. ```text Project base (Git HEAD or folder copy) --+ @@ -358,7 +380,9 @@ The modules below live in the skill's [`scripts/`](../plugin/skills/behavior-dif - **Model-based interpretation:** `decisions.py` reads the task, trial actions, final answers, numbered instruction-diff hunks, frozen purpose, and bounded - recorded returns. The same extraction call defines observable target criteria, + recorded returns. Extractor prompts travel over stdin, not command-line + arguments, so large comparisons do not hit the operating system's per-argument + size limit. The same extraction call defines observable target criteria, required evidence, and per-trial outcomes (`met`, `not_met`, `uncertain`, or `unassessable`), alongside choices, primary task result, narrative, and attention. Criteria cover every purpose goal and use the same question on both sides. diff --git a/plans/2026-10-19-intention-driven-preparation-implementation.md b/plans/2026-10-19-intention-driven-preparation-implementation.md index 0ae3481..6768578 100644 --- a/plans/2026-10-19-intention-driven-preparation-implementation.md +++ b/plans/2026-10-19-intention-driven-preparation-implementation.md @@ -1,10 +1,12 @@ # Implement intention-driven scenario and test-data preparation Date: 2026-10-19 -Status: PR 1 implemented and verified locally; draft PR delivery in progress. -Next delivery: finish PR 1 draft review, then implement PR 2's intention-driven -preparation. Report assessment/UI work remains deferred until both are complete -and verified. Do not repeat the public-repository survey or retune old cases. +Status: PR 1 merged as [#49](https://github.com/spacedock-dev/behavior-diff/pull/49). +PR 2 is published as [#50](https://github.com/spacedock-dev/behavior-diff/pull/50). +Preparation checks, both native-host smoke paths and independent source/evidence +reviews passed the bounded acceptance gate. Merge awaits corrected dogfood CI. +Report assessment/UI work remains deferred. Do not repeat the public survey or +retune old cases. ## 1. User request and product decision @@ -392,9 +394,9 @@ only after observing results from the completed preparation workflow. | Stage | Deliverable | Dependency | Status | |---|---|---|---| -| PR 1 | Deliver prepared shared data to real Before/After snapshots safely | None | Implemented; checks and independent review passed; draft PR delivery in progress; live verification pending | -| PR 2 | Generate and preflight scenario/data from intent, then execute and explain evidence | PR 1 | Planned; depends on PR 1 delivery | -| Preparation acceptance gate | Verify the integrated workflow against AC1–AC10 | PR 2 | Pending; required before declaring preparation complete | +| PR 1 | Deliver prepared shared data to real Before/After snapshots safely | None | Merged as #49; deterministic checks, independent review and corrected dogfood CI passed | +| PR 2 | Generate and preflight scenario/data from intent, then execute and explain evidence | PR 1 | Published as #50; independently approved; corrected dogfood CI remains the merge gate | +| Preparation acceptance gate | Verify the integrated workflow against AC1–AC10 | PR 2 | Passed at bounded two-host evidence level; limitations retained below | | Report design checkpoint | Decide what actual results need the report to communicate | Preparation acceptance gate | Deferred; requires user design review | | Later report PR(s) | Implement only the agreed assessment/presentation changes | Accepted report design | Not scoped or authorized by this plan | @@ -445,26 +447,27 @@ the affected architecture section and README usage explanation. **Not included:** autonomous scenario generation, report changes, a new fixture DSL, generic tool simulator, public CLI mode, or multi-file historical replay. -**Local implementation and verification record** - -- Added private `prepare_shared.py` staging/verification and the canonical - delivery reference. Integrated them into the skill without changing the public - runner interface or implementing the PR 2 generator. -- `python3 tests/shared-data-test.py --runner-smoke -v`: 28 checks passed. - Four successful runner scenarios exercised 16 synthetic trial processes across - Claude Code and Codex, including normal counts, explicit fast mode, absent - Before and explicit Before. Inspected actual trial files and captured host - reads, private-context exclusions, source provenance and original state. -- Boundary regressions cover frozen-input changes, unsafe paths and collisions, - unsupported project state, Git environment redirection and archive transforms. - Unsafe Git overrides and archive changes were reproduced before their fixes; - guarded handoff now rejects them without launching a host. -- Required deterministic suite, Bash/Python formatting and syntax checks passed. - Two independent read-only reviewers approved after material findings were fixed. - CI runs only the helper boundary suite; the runner smoke remains manual. -- No model calls or real Recce-Team replays were run. This establishes PR 1's - delivery mechanics, not automatic scenario quality or detection of invalidated - change assumptions. PR 2 and its acceptance gate remain pending. +**Delivery and verification record** + +- Merged as [#49](https://github.com/spacedock-dev/behavior-diff/pull/49). + Added private `prepare_shared.py` staging/verification and the canonical + delivery reference without changing the public runner interface. +- The final PR 1 shared-data suite passed 31 checks with `--runner-smoke`. + It exercised actual snapshots, normal and explicit fast counts, absent and + explicit Before, private-context exclusion, provenance and original state. + Boundary regressions cover unsafe paths, unsupported state, frozen-input + changes, Git environment redirection and archive transformations. +- A late CI failure exposed nested comparisons sharing their orchestrator's + runs store. Every trial now inherits its own `BEHAVIOR_DIFF_HOME`; actual + nested-run smoke tests verify the isolation. The old CI logs were unavailable, + so nesting was not established as that particular run's historical cause. +- Required deterministic checks and independent reviews passed. The corrected + dogfood run completed six trials, extraction and report publication. + CI's ordinary helper checks remain model-free; runner smoke remains manual. +- Later approved historical replays inspected raw evidence. The single-file + replay exercised the new delivery helper; a multi-file replay was explicitly + a historical control, not helper acceptance evidence. Neither reused fixture + proves autonomous generation. That proof belongs to PR 2. ### PR 2 — Prepare scenario and data from intent @@ -486,7 +489,8 @@ data, preflight/freeze them, run the comparison, and explain the observed outcom capabilities visibly blocked instead of silently narrowing the experiment. - Update the skill's result explanation to cover condition reach, unmet After intent, interruptions and raw-evidence versus automatic-report differences. - Do not change the report renderer, extractor or report schema in this PR. + Do not redesign the report renderer, extraction semantics or report schema. + The later user-authorized CI correction below changes prompt transport only. - Update the full workflow/flowchart and product explanation in the same PR. Add or adjust deterministic tests only for changed behavioral contracts. @@ -509,6 +513,73 @@ the real generator exposes a delivery defect. artifacts outside tracked files; public evidence summaries must be synthetic or sanitized and must not imply stronger findings than the records support. +**Local implementation and live evidence** + +- Canonical skill and its reference now own intent-driven preparation and + per-case raw assessment. README and architecture describe that workflow. + Delivery scripts, report schemas and renderer are unchanged. + A post-publication CI correction sends Claude extraction prompts over stdin + instead of argv to avoid Linux's per-argument size limit. Prompt contents, + models, extraction semantics and failure handling remain unchanged. + Removed affected wording-only assertions instead of pinning the new prose. +- Required deterministic suites passed, including 31 shared-data checks with + runner smoke. Repository Bash/Python formatting, changed-shell syntax and + ShellCheck passed. The report contract passed again after the final assertion + removal. +- A fresh Codex session received only a synthetic local invoice-review edit, + with no incident, task or dataset. It generated a neutral task, four common + data files, private goals/case criteria, capability probes and a frozen stage. + One ordinary match, a previously paid invoice and an ambiguous match shared + one coherent batch. The operator audited the inputs and additionally proved + read/CSV/arithmetic/scratch-write capabilities under native trial settings. +- The actual runner consumed those generated inputs in three Before and three + After attempts. All six initial snapshots had identical common files and + the correct instruction version; private criteria were absent, all six + traces recorded the relevant source rows, and the original project remained + unchanged. Trial selection resolved to `gpt-6.1-sol`; extraction resolved to + `gpt-6-luna`. Preparation requested `gpt-6.1-sol` separately. +- Five attempts completed: two Before and three After. All five recommended + the ordinary match, excluded the paid invoice and held the ambiguous match. + The remaining Before attempt failed with a model-capacity error after + reading the records. Its outcomes are unassessable, not evidence that the + instruction edit helped. No trial was retried or model substituted. +- The unchanged automatic report counted the interrupted attempt as valid and + could not assess source agreement despite the raw traces containing the + source rows. This is a report/normalization limitation, not a generated-data + defect. The skill now explicitly checks CLI status and native terminal events + before treating normalized text as completion. Report fixes remain deferred. +- A separate native Codex continuation executed the revised skill's result + explanation against the frozen notes and saved evidence. It read native + terminal events, identified the capacity interruption and both report gaps, + and assessed condition reach separately from outcomes for every case. + It found no contradicted goal among completed attempts and made no causal + improvement claim. This was host explanation, not another extraction or trial. +- Independent round one found a missing historical isolation branch. After the + frozen local comparison ended, the skill/reference were corrected to require + fresh context before historical purpose derivation when later outcomes are + already known. The local-edit smoke does not exercise that historical branch. +- Claude Code initially stopped before its first model call because authentication + had expired. That startup record was preserved. After the user restored native + authentication, a fresh Claude preparation generated its own neutral task, + three CSV files, seven cases and private criteria without an incident or dataset. + Three ordinary cases and four compatible boundaries shared one batch. +- All six Claude trials completed on `claude-opus-5-5`, with default Sonnet + extraction. Actual snapshots contained identical common data, correct + instruction versions and no private criteria; raw returns contained the sources. + All six recommended the same three invoices totaling $9,315.50 and withheld the + four questionable invoices. No payments occurred; this shows no added decision + benefit from the edit on these inputs. +- Native Claude result explanation inspected raw evidence and identified report + gaps. It also corrected a frozen preparation-note assumption: Write/Edit tools + were available, although unused. The observed read-only actions support local + behavior, not a claim that host permissions made writes impossible. Its use of + “paid” in summary wording means recommended payment, not an executed action. +- Codex preparation/execution used the pre-historical-isolation-fix snapshot; + its later explanation and the entire Claude smoke used the corrected source. + Both smoke cases are local edits, not live coverage of historical isolation. + Two independent reviewers approved the source in round two and the final + two-host acceptance evidence in round three, including these limitations. + ### Acceptance ownership | Criterion | Primary delivery and proof | @@ -524,6 +595,10 @@ the real generator exposes a delivery defect. | AC9 — freeze and preserve first results | PR 1 delivery freeze; PR 2 full preparation/execution discipline | | AC10 — skill prepares, runner consumes | PR 2 integrated smoke using PR 1's proven delivery path | +The two local smoke paths supply bounded evidence for AC1–AC10, not general +reliability or historical-isolation coverage. Preserve the interrupted Codex +attempt, initial Claude authentication failure and permission-note correction. + ### Deferred report design checkpoint After preparation passes the acceptance gate, review the generated scenario, diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index 553cce4..34fb531 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -12,13 +12,14 @@ no automatic verdict. The user judges the evidence. ## Ownership -The skill owns judgment. It finds the change, freezes its purpose, drafts the -decision-moment task, reviews any shared trial data, selects the current trial -stack and model, and explains the evidence. +The skill owns judgment. It finds the change, freezes its purpose, prepares +the scenario, shared data, and private assessment criteria from the change's +intent, preflights and freezes the inputs, selects the current trial stack and +model, and explains the evidence against each goal and case. -The scripts own repeatable mechanics. They stage and verify reviewed shared +The scripts own repeatable mechanics. They stage and verify prepared shared data, build variants, run trials, normalize traces, grade completeness, extract -decisions, and render the report. +decisions, and render the report. No script generates a scenario or data. The runner is bundled with this skill: `scripts/behavior-diff.sh` inside this skill's base directory. Pass `--agent` for the stack under test: @@ -74,6 +75,8 @@ the single-role or two-agent path. Create the fixtures with candidate: use it. Several: ask the user which one. None: explain that the change must exist as an uncommitted edit first, and stop. Then read `git diff -- ` so you understand what rule changed. + Read the selected instruction and its relevant unchanged references before + deriving the intent; a diff alone may omit behavior that must be preserved. Three cases have no git "before", and each still works: - **Untracked file** (git does not know it — e.g. `~/.claude/CLAUDE.md`): @@ -95,6 +98,12 @@ the single-role or two-agent path. Create the fixtures with In these cases read the change with `diff ` instead of `git diff`. + **Historical context boundary.** Before deriving historical purpose, read + the source-isolation rule in `references/scenario-and-test-data.md`. If this + session already knows later history or previous trial outcomes, purpose + derivation and preparation must use the fresh context described there. + If that isolation is unavailable, stop; disclosure alone is not a substitute. + **Freeze the purpose before drafting or running trials.** Derive concise, distinct goals from the current conversation and the selected diff without asking the owner to restate their reason. Use one or two sentences per goal; @@ -192,30 +201,37 @@ the single-role or two-agent path. Create the fixtures with `CI_COMMIT_SHA`, or `CI_COMMIT_BEFORE_SHA` for the actual comparison inputs. Missing PR/MR metadata is omitted; no API lookup or environment dump is needed. -2. **Draft the task.** `--task` must recreate the decision moment governed by - the changed rule, whether the goal is correction, addition, or preservation: - - Ask the user for the real request from the incident that motivated - the rule, and reuse it when they have one. - - Never leak the expected behavior into the task. The changed rule must - be the only place the guidance exists — if the task itself says what - "good" looks like, both variants will comply and the diff shows - nothing. - - When the rule fires mid-task, start the task at the decision point: - "You already did X (it is in the tree); decide/report Y." - - **Shared data.** If the decision moment needs common files that the - project base does not contain, read - `references/scenario-and-test-data.md` inside this skill's base - directory before running. Git trials start from HEAD, so additional - untracked or uncommitted data in the user's checkout never reaches them. That - reference delivers files the user supplied or approved to every trial. - It does not generate a scenario or data. - -3. **Run it as soon as the task is known and any shared data is verified.** - Do not ask the user to confirm the file, task, cost, or run mode. Do not - mention trial counts, cost, or full versus fast modes during normal - execution. +2. **Prepare the scenario and data.** Before drafting anything, read + `references/scenario-and-test-data.md` inside this skill's base directory + and follow its preparation recipe. In short: + - From the frozen goals and the selected diff, list the new and preserved + behavior, the observable capabilities it needs, and the environment + premises those capabilities depend on. + - Write a neutral `--task`: a natural request that starts before the tested + work and reaches the changed rule. It must not state the expected answer, + the rule, or a historical mistake; the changed instruction is the only + place the guidance exists. + - Reuse project data when it already exercises the capabilities. Otherwise + prepare one coherent dataset of ordinary inputs plus a bounded set of + compatible edge cases that challenge the goal's assumptions. + - Keep per-case criteria, mappings, and omissions in private notes, never + in the task or trial files. + + Do not require an incident, a real past request, or an existing eval from + the user. Ask only when a material requirement cannot be reached from the + change, project, or conversation. For a historical comparison, prepare from + the contemporary change only, before reading later fixes or results. + +3. **Preflight, freeze, then run.** Run as soon as the reference's preflight + passes and the task, data, criteria, and settings are frozen. Do not ask the + user to confirm the file, task, cost, or run mode. Do not mention trial + counts, cost, or full versus fast modes during normal execution. First satisfy the privacy and sensitive-content approval boundary above; - automatic execution never overrides it. + automatic execution never overrides it. If the planned environment cannot + support a required capability or premise, stop and report it; do not silently + substitute a narrower test or fabricate approvals. Stop preparation workers + before launch, and keep the first result: no retuning or reruns to manufacture + a finding after seeing outcomes. Under Claude Code or Codex, preserve the current stack: @@ -252,8 +268,9 @@ the single-role or two-agent path. Create the fixtures with does not look anything up. Never put source details in `--task`, a trial prompt, or a sandbox file. - **Shared data.** When step 2 selected shared data, do not run the commands - above from the user's checkout. Prepare and verify the stage with + **Shared data.** When step 2 prepared new shared data, or private context + in the checkout cannot otherwise be kept from trials, do not run the + commands above from the user's checkout. Prepare and verify the stage with `references/scenario-and-test-data.md`, then run the same runner command from `$ROOT/project` with `$ROOT/home` as `BEHAVIOR_DIFF_HOME`, exactly as that reference shows. Keep the same agent, model, `--fast`, purpose, @@ -296,6 +313,14 @@ the single-role or two-agent path. Create the fixtures with whether the situation was reached. Do not infer a missed situation from identical behavior alone, or claim that the rule generally works or fails. + Then assess each private case from raw evidence (actions, tool returns, + artifacts), not only final answers, as the reference describes. Separately + record whether its condition was reached, missed, or unknown and whether its + intended outcome was supported, contradicted, or lacked sufficient evidence. + Summarize each goal without hiding missed cases. Report interrupted or + unassessable attempts and harness failures separately from requirement failures. + Identical Before and After behavior can still fail a goal After added. State + where your assessment agrees with or differs from the report; do not edit it. Orient the reader briefly: say what the agent was asked to do, the changed skill's role when relevant, and the decision point tested. Lead with the practical supported contrast, not an internal workflow label. At first use, diff --git a/plugin/skills/behavior-diff/references/scenario-and-test-data.md b/plugin/skills/behavior-diff/references/scenario-and-test-data.md index 7108ae3..17e277f 100644 --- a/plugin/skills/behavior-diff/references/scenario-and-test-data.md +++ b/plugin/skills/behavior-diff/references/scenario-and-test-data.md @@ -1,10 +1,140 @@ # Scenario and test data -Read this file before you run a comparison whose task needs shared files that -the runner's project base does not contain. It explains how to deliver an -already selected dataset to every Before and After trial. It does not generate -a scenario or data. Step 2 of `SKILL.md` still drafts the task, and the user -supplies or approves the files. +Read this file before drafting the task for any comparison. It is the single +recipe for preparing the scenario, shared data, and private assessment +criteria from the change's intent, for preflighting and freezing them, and for +delivering shared files to every Before and After trial. You, not a script, +prepare the scenario and data; `prepare_shared.py` only stages and verifies +files you prepared and reviewed. + +Spacedock workflow comparisons use the fixtures in +[spacedock-duo.md](spacedock-duo.md) for delivery; still derive their task and +criteria with the recipe below. + +## Prepare from intent + +Use the contemporary change, relevant unchanged instructions and references, +and available owner context. A supplied real request is usable, but not required. +For a historical comparison, exclude later fixes, later incidents, earlier +fixtures, and trial results; do not mine host transcripts. If the orchestrator +already knows later history or previous outcomes, derive historical purpose and +prepare the inputs in a fresh context that inherits none of that conversation. +Supply only the selected contemporary instruction versions, required references, +available contemporary owner context or commit message, and this generic method. +Use a supported fresh session or isolated subagent; if unavailable, stop and +report the missing isolation prerequisite before deriving or freezing purpose. +If case selection or tooling was informed by hindsight, still disclose that +limit: isolation does not remove historical selection bias. + +1. **Goals.** Start from the frozen purpose goals. Name new behavior and + behavior that must survive the change. Mark preservation goals as inferred + unless stated. +2. **Capabilities and premises.** For each goal, list what the agent must + observe, decide, or produce. For each capability, list the environment + facts it depends on (files, query results, tools, permissions, repository + state, time), where the trial can reach each fact, and which source is + authoritative. Assessment-only knowledge must not stand in for a premise + missing from the trial. If an outcome depends on time, set one as-of + timestamp and timezone, give both variants the same neutral as-of statement, + and make every simulated clock and calculation agree. +3. **Task.** Write a natural user request that starts before the tested work + and reaches the changed rule. It defines starting state and scope only. It + must not state the expected answer, the rule, a findings list, or a + historical mistake. If both variants would comply because the task says + what good looks like, rewrite it. Start mid-workflow only when skipped work + is outside the capabilities being tested and the starting state is coherent. +4. **Data.** If the project base already exercises the capabilities, reuse it + and add nothing. Otherwise build one coherent dataset: ordinary inputs plus + a bounded set of plausible edge cases that challenge the goals' + assumptions. Choose cases by relevance, plausibility, assessability, and + added coverage of the intent; do not fill a quota or add redundant cases. + Combine compatible cases; cases that contradict or obscure each other need + separation, which is a separate comparison, not an incoherent fixture. Let + the agent acquire facts through supported files or queries; do not replace + discovery with a prepared findings list. Query returns can contain legitimate + task facts, but assessment conclusions stay private. Any local tool or adapter + must accept both correct and incorrect actions, record the actual submitted + arguments or artifacts, and not enforce the desired result. Extra scenarios + or trial sets require separate approval. +5. **Private notes.** Outside every trial input, in the private `0700` + directory, record with mode `0600`: the goal → capability → premise → + case mapping; why each case was selected; evidence that establishes whether + each condition was reached, missed, or remains unknown; separate criteria for + supported, contradicted, or insufficient-evidence outcomes for every case; + cross-case cues and interactions; capabilities left untested; task wording or + tool design that makes one action easier; and deviations from the original + project, including replaced base files. Combined cases are not independent + experiments. These are notes, not a schema, and never enter trials, the task, + or `--purpose-file`. + +Ask the user only when a material requirement cannot be reached from the +change, project, or conversation. Do not require an incident or an existing +eval. + +## Preflight + +Before spending trial calls, check the actual environment the trials will use, +under the host's intended permissions: + +- The selected After bytes and the Before content or verified absence are + available; every reference required by either version is in the trial base. + Exactly the selected instruction differs between variants. +- Exercise the required reads, queries, local tests, artifact writes and safe + action recording in scratch copies under the intended host tools and + permissions. Confirm consistent facts and authoritative sources, including + time when needed. Do not assume subagent dispatch or approvals exist. +- Represent relevant branch/remote state, permissions and existing user work + explicitly. Use safe local artifacts for modeled publication; never include + credentials or perform real external effects merely to test an instruction. +- Private context (purpose, source, notes, criteria) must not enter the trial + base. The runner removes the specified purpose and source files, but does + not find arbitrary evaluator files. Stage with `--exclude` for other private + paths; use reviewed existing task data when no new data is needed. If safe + staging is unsupported, stop rather than silently leak. + +Run probes in scratch copies only. A probe may exercise discovery queries to +check the environment, but must not deposit its findings in trial inputs or +change the original project or frozen stage. The trial still performs discovery +itself. If a required capability or premise is missing, stop and report it. +Do not substitute a generic simulator or fabricate approvals. A deliberately +modeled limitation is part of the scenario; an accidental harness failure is not. + +## Freeze + +Before launch, freeze privately: the task text, purpose, source, data or +stage, required references, tool behavior, private notes and criteria, the +agent and model, extraction host and model, reasoning settings when known, +`--fast` choice, and the preparation model. Record a readable inventory with +digests (for example `shasum -a 256`) in the private directory. Do not infer +reasoning effort from a model name. + +`prepare_shared.py verify` freezes only the staged project and Before. It does +not hash the task, purpose, source, notes, or tooling; recompute your +inventory digests immediately before launch and stop if any differ. + +Keep the default three trials per side, or `--fast` only when the user asked. +More cases do not add runs; a richer scenario may still use more tokens per +trial. Stop preparation workers before launch. Keep the first result. Do not +change cases, data, or criteria after seeing outcomes, and do not rerun to +produce a finding. + +## Assess the result + +After the run, read raw trial evidence: actions, tool returns, and produced +artifacts, not only final answers. For every case in every trial, separately +record whether the condition was reached, missed, or unknown and whether its +intended outcome was supported, contradicted, or lacked sufficient evidence. +Then assess each goal without hiding missed cases or minority outcomes. +Check each trial's status, CLI exit and available native host terminal events +before treating normalized final text as a completed answer. A failed turn can +leave only an earlier progress message in that field. Keep interruptions, +provider errors and harness failures separate from requirement failures. +Nonempty error or progress text is not completion; a child's successful return +does not prove its caller finished the workflow. +Before and After can behave identically while After still fails its new goal; +report the unmet After goal alongside the unchanged behavior. State where this +raw assessment agrees or disagrees with the automatic report, and disclose +untested capabilities. Never edit an existing report to match the audit. ## When to deliver shared data @@ -12,13 +142,12 @@ The runner builds each trial from the project base. In a Git project, the base is `git archive HEAD`. Outside Git, the base is a copy of the folder. Apart from the selected instruction, untracked and uncommitted Git files do not enter trials. -Use shared-data delivery only when all of these conditions are true: +Use shared-data delivery when the comparison changes exactly one instruction +file, every Before and After trial must receive the same files, and either the +task needs files that are not in the project base or private context in the +project base must be kept out of trials. -- The task needs common files that are not in the project base. -- Every Before and After trial must receive the same files. -- The comparison changes exactly one instruction file. - -If the task needs no extra files, use the normal step 3 command instead. +If neither applies, use the normal step 3 command instead. Do not use this path for: @@ -41,9 +170,9 @@ reach the trials. evaluator notes. Review every file before preparation. The helper rejects unsafe file types and paths. It does not scan contents for secrets or private text. - Shared paths replace matching base files. Record the intended deviations - from the original project, including any replaced base files, in private - preparation notes outside the trial inputs before staging. + Shared paths replace matching base files; record them in the private + notes. When staging only to exclude private context, the data directory + holds reviewed existing project data that the task needs. 2. **Private context.** Keep the purpose file, the source file, and evaluator notes in a private directory with mode `0700`, outside the project, the data directory, and the stage. Give each file mode `0600`. If the project already @@ -179,10 +308,10 @@ local names, not original revisions. ## Freeze discipline -- After `verify` passes, treat the stage, task, purpose, and source as frozen. - Stop any preparation worker before launch. No agent may edit - `$ROOT/project/`, `$ROOT/before`, `$ROOT/home/baselines/`, `freeze.json`, or - the runner and its scripts. +- After `verify` passes and your inventory digests match, treat the stage, + task, purpose, source, notes, and settings as frozen. See [Freeze](#freeze). + No agent may edit `$ROOT/project/`, `$ROOT/before`, `$ROOT/home/baselines/`, + `freeze.json`, or the runner and its scripts. - To change the data, prepare a new stage and run a new comparison. Never prepare a new stage or edit data to turn a negative or inconclusive result into a finding. Keep and report the first results. diff --git a/plugin/skills/behavior-diff/scripts/decisions.py b/plugin/skills/behavior-diff/scripts/decisions.py index 73b1023..be618a1 100755 --- a/plugin/skills/behavior-diff/scripts/decisions.py +++ b/plugin/skills/behavior-diff/scripts/decisions.py @@ -1030,7 +1030,8 @@ def _codex(prompt, model): def _claude(prompt, model): proc = subprocess.run( - ["claude", "-p", prompt, "--model", model, "--allowedTools", ""], + ["claude", "-p", "--model", model, "--allowedTools", ""], + input=prompt, capture_output=True, text=True, ) diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 41560d7..f140aa5 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -142,18 +142,6 @@ reject_output 'capsule' "$skill" \ reject_output 'capsule' "$spacedock_reference" \ 'Spacedock reference still uses the unexplained capsule term' require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; } -require_output 'The skill owns judgment.' "$headless_skill" \ - 'headless skill does not state its judgment ownership' -require_output 'The scripts own repeatable mechanics.' "$headless_skill" \ - 'headless skill does not state script ownership' -require_output '--agent pi' "$headless_skill" \ - 'headless skill does not name the Pi trial stack' -require_output '' "$headless_skill" \ - 'headless skill does not require the exact Pi model' -require_output '--agent omp' "$headless_skill" \ - 'headless skill does not name the OMP trial stack' -require_output '' "$headless_skill" \ - 'headless skill does not require the exact OMP model' require_output 'Pi has no built-in subagent dispatch.' "$skill" \ 'live skill invents a built-in Pi dispatch path' require_output "one \`task\` batch" "$skill" \ @@ -168,33 +156,6 @@ require_output 'claude, codex, pi, or omp' "$codex_manifest" \ 'Codex manifest does not name all trial stacks' require_output 'Pi and OMP are trial stacks, not plugin hosts' "$readme" \ 'README does not separate trial stacks from plugin hosts' -require_output "Only add \`--fast\` when the user explicitly requested it" \ - "$headless_skill" \ - 'headless skill does not reserve fast mode for explicit requests' -require_output 'full versus fast modes' "$headless_skill" \ - 'headless skill still exposes implementation modes to the user' -reject_output 'Confirm before running' "$headless_skill" \ - 'headless skill still asks for run confirmation' -reject_output 'plus the cost:' "$headless_skill" \ - 'headless skill still advertises model-run cost' -require_line ' behavior-diff.sh --agent --file --task "" --purpose-file ' \ - "$headless_skill" \ - 'default command does not preserve the current agent host' -require_line ' behavior-diff.sh --agent --file --task "" --purpose-file --fast' \ - "$headless_skill" \ - 'explicit fast command does not preserve the current agent host' -require_output 'Freeze the purpose before drafting or running trials.' "$headless_skill" \ - 'headless skill does not capture purpose before trial design' -require_output 'current context, including supplied summaries; do not open host transcript' "$headless_skill" \ - 'purpose capture invents host transcript access' -require_output 'source:"diff", basis:"inferred"' "$headless_skill" \ - 'headless skill has no labeled diff-only fallback' -require_output 'never revise goals after seeing Before/After results' "$headless_skill" \ - 'headless skill does not freeze purpose independently of outcomes' -require_output 'Conservative exclusions' "$headless_skill" \ - 'headless skill does not qualify privacy exclusions' -require_output 'specific approval for that content or a safe fixture' "$headless_skill" \ - 'headless skill lacks the sensitive-content consent boundary' require_output 'Run behavior-diff with this exact task:' "$demo_skill" \ 'demo journey does not supply the exact fixture task with the nudge response' require_output 'the exact task step 1 printed' "$demo_skill" \