[kentwelcome]

What changes

Behavior Diff can now start from a local instruction edit without asking the user for an incident or prepared evaluation dataset. The skill derives the intended change and behavior to preserve, selects observable capabilities, writes a neutral task, and prepares coherent ordinary and boundary-case data. It preflights and freezes the inputs before handing them to PR #49’s existing delivery helper.

After execution, the skill assesses each case from raw evidence: was the relevant condition reached, was the intended outcome supported, and did the attempt actually complete? It explains disagreements with the automatic report instead of silently changing that report. Historical preparation must use an isolated context when later outcomes are already known.

This is PR 2 of the intention-driven preparation plan. Preparation remains agent-owned; there is no new generator service, public schema, simulator framework, or runner API. README and architecture describe the new workflow. Wording-only assertions were removed rather than repinned; deterministic runtime contracts remain.

Verification

• Required deterministic suites passed, including 31 shared-data boundary/runner-smoke checks, hook/model/purpose contracts, report/provenance/CI-context checks, release checks, and human-evaluation helper checks.
• Repository shfmt and Ruff checks, changed-shell syntax/ShellCheck, and git diff --check passed.
• Fresh native Codex and Claude sessions each generated their own synthetic scenario, data and private criteria from a local instruction edit. No incident, task or dataset was supplied. Actual trial snapshots and raw source returns were audited for input equality, instruction fidelity and private-criteria exclusion.
• Codex: gpt-6.1-sol trials and gpt-6-luna extraction. Five of six attempts completed with unchanged correct decisions; one Before attempt stopped on provider capacity and remains unassessable. No trial retry or model substitution.
• Claude: all six attempts completed on claude-opus-5-5, with default Sonnet extraction. Both versions recommended the same three invoices and withheld four questionable ones. No payment occurred; no improvement attributable to the edit is claimed.
• Native result-explanation continuations inspected raw evidence and identified automatic-report gaps. Two independent read-only reviewers approved the complete source and bounded two-host acceptance evidence after the historical-isolation finding was fixed.

Limits and deferred work

• Report UI, extractor, schema and runtime normalization are unchanged. Current reports can omit recorded source evidence or mistake an interrupted attempt’s progress text for a final answer; raw explanations disclose these gaps.
• A Claude preparation note incorrectly assumed Write/Edit tools were unavailable. The audit corrected that assumption: tools were available but unused. Observed read-only behavior is not enforced containment.
• Codex preparation/execution preceded the historical-isolation correction; its later explanation and the entire Claude smoke used corrected source. Neither local-edit smoke exercises historical isolation.
• Synthetic cases establish bounded preparation/consumption evidence, not exhaustive coverage, unknown-side-effect detection or general reliability. Private artifacts remain outside the repository.

Tracking: https://linear.app/recce/issue/DRC-4917
Plan: plans/2026-10-19-intention-driven-preparation-implementation.md

Draft for review; do not merge implicitly.

[kentwelcome]

Independent AI review record

Reviewed base: 8cccfec08faa84eddbd3905e7827696328f0f2fc (#49).
Published head: c5ca36e55995ce4116b3193269887958efbdaf09.

The following are independent read-only AI subagent reviews, not human GitHub approvals. Reviews covered all six changed files. Reviewers did not run tests, formatters or live models; the implementing agent supplied verification and reviewers inspected evidence.

Round 1 — REVISE (Standards and Spec)

Both reviewers identified the same P2 finding: historical preparation must move to fresh context before deriving purpose when the orchestrator already knows later outcomes. Merely disclosing hindsight did not satisfy the accepted plan. Corrected the canonical skill and reference to require isolation, specify contemporary inputs, and stop when isolation is unavailable. Selection-bias disclosure remains required.

Decisions required: none. Optional suggestions: none. Scope audit: no new runtime infrastructure, public schema, simulator or report redesign; affected wording-only assertions removed rather than repinned.

Round 2 — APPROVE source (both reviewers)

Both reviewed the complete corrected scope and confirmed the historical-isolation finding resolved. They also approved native-status/terminal-event checks in raw assessment. Source approval did not waive the then-blocked Claude authentication/live gate. Codex generation, actual input consumption and raw explanation were inspected; its provider-capacity interruption remained unassessable, not an instruction failure.

Material findings, decisions and optional suggestions: none. Scope remained the approved PR 2 boundary.

Round 3 — APPROVE source and bounded two-host evidence (both reviewers)

After native Claude authentication was restored, both reviewed its generated preparation, actual six-trial snapshot audit, model records and raw explanation. The bounded AC1–AC10 gate is satisfied across both hosts. The permission-note correction and recommendation-versus-payment distinction are explicitly preserved. No general reliability, causal improvement, permission-enforced containment, or live historical-isolation coverage is claimed.

Material findings, decisions and optional suggestions: none. No unauthorized mechanisms or scope changes. Publication and merge were not authorized by reviewer approval; the owner subsequently requested this draft PR, not a merge.

Verification supplied by the implementing agent

Required deterministic suites passed, including 31 shared-data boundary/runner-smoke checks. Repository shfmt/Ruff and changed-shell syntax/ShellCheck passed; final git diff --check passed. Fresh native Codex and Claude preparations generated their own scenarios/data; actual trial snapshots and source returns were audited. Codex had five completed attempts and one capacity interruption; Claude completed six. Both native explanation continuations distinguished raw evidence from current automatic-report gaps.

No saved report was rewritten. Private artifacts were not committed. The source-version chronology and remaining evidence limits are retained in the implementation plan and PR description. Published head matches the approved source changes and factual acceptance record. CI has started; this comment does not claim CI completion.

[github-actions]

Behavior Diff · plugin/skills/behavior-diff/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — plugin/skills/behavior-diff/SKILL.md at merge-base 8cccfec08faa84eddbd3905e7827696328f0f2fc.
• After — plugin/skills/behavior-diff/SKILL.md at PR head 173e35d5776eae3b29cc70cc45e49866a3e7e18c.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-10-26T08:17:26Z). Evidence ZIP.
• Workflow run — run 39221048988, attempt 1.

Summary

Before made data approval a blocker in 3 of 3; After did not

1. What this edit was meant to change

Model summary of the recorded goals

When starting from a local instruction edit, the skill should build its own neutral task, mixed ordinary and edge-case data and private criteria without asking for an incident. It should stop if the environment can't support the plan, and read the instruction file before deriving intent.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. What the evidence shows

The agent applied a rule edit and prepared a behavior comparison up to launch, drafting its own task and CSV data.

Based on final answers. An answer describing an action does not prove it happened.

Before

Observed start: Each trial drafted its own synthetic CSV

From a recorded command, before the final answer. A recorded command does not establish successful completion or cause.

↓

CSV approval as blocker

Listed CSV approval as a blocker before launch.

3 of 3 trials

After

Observed start: Each trial drafted its own synthetic CSV

From a recorded command, before the final answer. A recorded command does not establish successful completion or cause.

↓

Approval for setup only

Asked approval of the fallback standalone setup, not the CSV.

1 of 3 trials

Items to review

Listed the CSV and setup as items to check, with no approval gate.

2 of 3 trials

Reported primary result — mixed

Before: Left verified stage and runner command (2 of 3 trials); Deleted stage; command depends on user option (1 of 3 trials)

After: Left verified stage and runner command (3 of 3 trials)

Based on reported final answers; these outcomes do not prove execution.

What this means: Both sides prepared their own task and data without asking for an incident. After asked for less data approval and wrote per-case private criteria, which Before's brief notes mostly lacked. No trials ran, so rule behavior is untested. Comparison 3 · Comparison 5 · Comparison 8

3. What needs your attention

No trials were launched, so the rule's effect is untested. In these trials, After wrote richer private criteria and asked for less data approval. Two After answers also flagged missing sections in the edited skill's reference file.

Edited skill points to reference sections that two answers say are missing

Two After agents wrote their own cases and criteria because the referenced recipe was not in the reference file. No Before trial reported this.

Relationship: Additional concern
Evidence: Observed difference · Final answers, not proof of execution

Limit: This comes from agent answers. The reference file contents were not captured, and After-2 did not comment on it either way.

What you can do: Check that references/scenario-and-test-data.md contains the preparation recipe and source-isolation rule the skill cites.

Before made CSV approval a blocker; After mostly did not

Every Before answer listed the CSV as needing approval. After answers listed the data as something to review, and After-2 asked approval only for its fallback setup.

Relationship: Related to the intended change
Evidence: Observed difference · Final answers, not proof of execution

Limit: Only three trials per side, all in one scenario with a synthetic CSV. Wording in final answers is not a recorded approval event.

What you can do: Decide whether a quick-look note on generated data is enough, or whether the skill should state explicitly that no approval is needed.

4. What we still cannot tell

Watch out: Only three trials per side, one scenario, and all findings come from final answers, not run results. Self-reported statements are not verified. Comparison 8 · Comparison 10

• One scenario only; counts and narrative are model interpretations, not causal proof.
• Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result.
• The primary result varies across trials; see its full distribution in Behavior diff.

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 173e35d5776eae3b29cc70cc45e49866a3e7e18c); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}

[kentwelcome]

Pre-merge check on head c5ca36e: Format and Unit passed, but Behavior Diff run 39217533591 failed its final gate. Downloaded result.json records 3/3 Before and 3/3 After complete, runner_exit 0, report_available true, status partial, extraction unavailable. Error: Decision extraction was unavailable; the report shows trial evidence without a decision diff. The job log and published bundle do not contain the underlying extractor failure diagnostics, so this is not yet attributable to provider capacity, authentication, malformed extraction, or the instruction change. No models were rerun and no code was changed. Earlier local acceptance remains valid but does not turn this CI gate green. Merge paused pending resolution or an explicit owner decision on this newly disclosed limitation.

[kentwelcome]

Review round 4 — CI extraction correction (not yet published)

Reviewed base: 8cccfec08faa84eddbd3905e7827696328f0f2fc.
Published head: c5ca36e55995ce4116b3193269887958efbdaf09.
Reviewed snapshot: published head plus the local decisions.py::_claude stdin correction and architecture explanation.

Finding and disposition

The failed dogfood run retained six graded-complete trials and reports, but no usable extraction. The Action did not retain the original extraction exception.

A conservative reconstruction from the retained commands, final answers, frozen instruction diff and fixed extraction instructions requires 134,036 bytes, excluding all tool-return text, purpose goals and style guidance. Claude extraction passed the whole prompt as one argv argument. Linux with 4 KiB pages limits one argument to 131,072 bytes, including its terminator.

A network-disabled Linux smoke using the actual _claude function reproduced errno 7: Argument list too long before its executable started. After passing the unchanged prompt through stdin, the same smoke delivered all 134,036 bytes and exited 0. This fixes a demonstrated pre-existing transport defect; the original traceback is unavailable, so it is not proof that no other failure occurred in that run.

No evidence trimming, retries, model changes, report-schema changes, or CI bypass. No changes to the external Action repository are needed.

Independent AI review

• Standards reviewer: APPROVE. No material findings or decisions required. Prompt/model/output/error semantics preserved; no unauthorized mechanism.
• Specification reviewer: APPROVE, source only. Full PR2 intent/preparation/isolation/raw-assessment scope remains compliant. Actual corrected dogfood CI is still required.
• Both suggested correcting stale publication/extractor wording in the plan. That documentation-only correction is now applied locally and is receiving final snapshot review.

These are independent AI subagent reviews, not human GitHub approvals. Reviewers ran no tests or model calls.

Gates

Failing-before/passing-after Linux transport smoke completed. Full deterministic checks are running. Code remains unpublished; actual corrected CI has not run. Merge remains blocked until that gate passes. The first failed run and its evidence remain preserved.

[kentwelcome]

Review round 5 — published correction and final source approval

Base: 8cccfec08faa84eddbd3905e7827696328f0f2fc.
Reviewed and published head: 173e35d5776eae3b29cc70cc45e49866a3e7e18c (DCO signed).

Both independent AI reviewers re-inspected the complete latest scope after the plan bookkeeping correction:

• Standards: APPROVE. No material findings, decisions required, or remaining suggestions.
• Specification: APPROVE, source only. No material findings, decisions required, or remaining suggestions. Actual corrected CI remains mandatory.

Scope audit: the existing Claude extractor now receives the unchanged prompt on stdin; no new dependency, schema, model, retry, trimming policy, public CLI mode, or gate bypass. The plan distinguishes published PR2 and the transport correction from deferred report redesign. These are AI subagent reviews, not submitted human approvals.

Verification completed by the primary agent

• Network-disabled Linux smoke: original transport failed with E2BIG; corrected transport delivered all 134,036 bytes and exited 0.
• decisions.py --check, hooks, live-report contract, source provenance, report source, CI context, report CI context, release workflow, and all three human-evaluation suites passed.
• Repository shfmt, Ruff formatting (140 files), and git diff --check passed.

The actual corrected dogfood run is now running against this exact head, with three trials per side and unchanged configured roles (opus trials/drafting, sonnet extraction). This is an infrastructure-fix validation run, not a retune of the earlier preparation acceptance experiment. The first failed run remains preserved. Do not merge before the corrected gate succeeds.

[kentwelcome]

Post-review evidence limitation — original dogfood scenario

Reviewed head remains 173e35d5776eae3b29cc70cc45e49866a3e7e18c (base 8cccfec08faa84eddbd3905e7827696328f0f2fc).

The original failed run's generated task directed every trial to shared /tmp/bd-prep/ paths. Recorded commands wrote those paths, and final answers disclosed concurrent writes and possible overwrites. That run is contaminated and is not independent or causal evidence that PR2 improves preparation. It remains preserved rather than discarded or reclassified.

Both independent Standards and Specification reviewers inspected this evidence and retained APPROVE, with no material findings, new decisions required, or additional mechanism warranted. This is an existing outside-worktree isolation limitation, not introduced by PR2; separate directories are not a filesystem sandbox. Dedicated native-host preparation smoke and snapshot audits remain separate acceptance evidence.

The contaminated records still establish prompt size; their contents' validity does not remove the demonstrated Linux transport defect. A green corrected dogfood run will establish pipeline operation only—not automatically independent causal effectiveness. Its evidence will be checked on its own merits. No sandboxing or task-drafter redesign is silently included in PR2.

[kentwelcome]

Final pre-merge gate — passed

Approved head: 173e35d5776eae3b29cc70cc45e49866a3e7e18c. Base: 8cccfec08faa84eddbd3905e7827696328f0f2fc.

Corrected dogfood CI succeeded in prepare, run and publish. Its retained result.json confirms this exact core/head revision, Before 3/3, After 3/3, extraction complete, runner 0, error null, and report available. Format and Unit checks are also successful; GitHub reports a clean, mergeable PR.

Both independent AI reviewers inspected the final result and report limitations and retained APPROVE, with no material findings, decisions required, or suggestions. No additional mechanism or scope expansion is required. Source has not changed since their approval.

Limits retained, not waived

• The Action changes only the selected SKILL.md, holding companion files at the merge base. Its missing-reference concern therefore does not describe the shipped bundle: the published reference contains the new recipe and historical-isolation sections. This CI run proves pipeline operation, not evaluation of the complete multi-file PR; dedicated native-host smoke/snapshot audits remain the integration evidence.
• Complete extraction does not mean every goal received target assessment; the report explicitly says target assessment is unavailable.
• The earlier failed run's shared-directory contamination remains recorded; it is not causal preparation-success evidence.

The user requested merge and explicitly required fixing, not bypassing, the failed gate. That operational gate is now satisfied. Merging the reviewed head without further code changes.