[kentwelcome]

Summary

PR 1 of the intention-driven preparation plan (DRC-4916): safely deliver prepared shared scenario data into real Before/After trial snapshots.

• Adds private prepare_shared.py and one canonical shared-data delivery reference, integrated into the skill without public runner interface changes.
• Adds deterministic boundary coverage and opt-in manual runner smoke; CI runs only helper boundaries.
• Updates architecture, usage, verification documentation and the implementation plan.
• Isolates nested comparison storage per trial across all hosts, and preserves default hook-baseline lookup for an empty home override.

Not included: automatic scenario/data generation (PR 2), multi-file delivery support, report assessment/UI changes.

Verification

• Required deterministic suites, Bash/Python formatting and syntax checks passed.
• Final manual fake-host runner suite: 31 tests passed, including real nested runner regressions for Claude Code and Codex, plus empty-home baseline coverage.
• Independent read-only Standards and Spec reviewers: APPROVE.
• GitHub CI Format and Unit: passed.

Requested private live verification

Two frozen historical scenarios were rerun once, without scenario regeneration or retries. Each used three Before and three After trials plus one extraction. Actual assistant-event model identities were checked, not merely command arguments.

• New-helper delivery case: anthropic/claude-opus-4-5-20251101, high reasoning, OMP. External shared data and historical instructions were split and reassembled through the new helper; resulting content/executable bits matched the frozen originals. All six trials completed. Raw evidence reproduced an unmet After verification requirement; the automatic report did not identify it.
• Historical control only: openai-codex/gpt-6.1-sol, high reasoning, OMP. The owner approved using the existing private multi-file harness because this case exceeds PR 1's single-file contract. All six trials completed; its previously observed output-duplication failure did not reproduce. This is not acceptance evidence for the new helper.

Extraction was separately invoked using each historical condition's same pinned model. Inputs, traces and reports remain private; no private source excerpts or report artifacts are attached. These runs do not prove PR 2 generation or automatic assumption validation.

CI failure resolved on the corrected head

The original dogfood job failed to collect a report. A deterministic nested-run reproduction demonstrated parent run-directory contamination; trial-local storage fixes that path. An independent review also found and verified the empty-home fallback correction. Both regressions failed before their fixes and passed afterward.

Fresh dogfood run 39213095873 used head/core 2e49baa409d18cec1c16c6dde22e656297f077dc. Its uploaded result confirms status complete, 3/3 Before and 3/3 After, extraction complete, runner exit 0, no error, and report available. Prepare, run, and publish all succeeded; Format and Unit also passed. Both independent reviewers approved this corrected scope.

The original logs were deleted, so nesting remains an unproven explanation of that historical occurrence. The isolation defect is reproduced and fixed, and current CI successfully collects/publishes its report. The private real-case results above predate the isolation correction; they were not rerun.

Linear: DRC-4916

[kentwelcome]

Independent review record (read-only, per REVIEWER_GUIDELINES.md): fresh Standards reviewer — APPROVE; fresh Spec reviewer — APPROVE. Scope: PR 1 shared-data delivery files at commit 6406504 on base 7f615f0. Earlier material findings were fixed before these approvals. Live verification pending (DRC-4916).

[github-actions]

Behavior Diff · AGENTS.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — AGENTS.md at merge-base 7f615f04ed84de645ea4a5896e64424609c54b6e.
• After — AGENTS.md at PR head 64065049f39f45cdd589a7b41a869abc2ec3f17b.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-10-26T04:18:51Z). Evidence ZIP.
• Workflow run — run 39198414302, attempt 1.

Summary

All three After runs tried the shared-data test; no Before run did

1. What this edit was meant to change

Model summary of the recorded goals

Agents running the required deterministic suite should also run the shared-data test, in its default form, while keeping the existing checks and their order, with the new test placed after the decisions check.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

Context for this comparison: AGENTS.md lists tests/shared-data-test.py, but After trials found no such file in the repo. Only that AGENTS.md line mentions it. Comparison 1

2. What the evidence shows

The agent ran the repo's required test suite before a PR. The edit added a shared-data test to AGENTS.md's list.

Based on recorded tool calls. Recorded calls do not establish successful completion.

Before

Test not attempted

The agent ran the 11 older commands and never invoked the shared-data test.

3 of 3 trials

After

Ran test; file missing

The agent ran the shared-data test third, in listed order, and got exit 2 because the file does not exist.

3 of 3 trials

Reported primary result — mixed

Before: Hold off: change not in tree (2 of 3 trials); Green, but confirm change present (1 of 3 trials)

After: Not ready: missing test and change (3 of 3 trials)

Based on reported final answers; these outcomes do not prove execution.

What this means: The edit made the agents attempt the new test in the listed order, as intended. The file is missing here, so this run could not show a passing result. Comparison 1 · Comparison 4

3. What needs your attention

Attention assessment unavailable.

No valid attention assessment is saved for these trials. This does not establish that there are no remaining or additional concerns. The original explanation and trial evidence remain available.

4. What we still cannot tell

Watch out: The missing file, not a test failure, caused the step to fail. All trials also saw a clean working tree. Comparison 1 · Comparison 2

• One scenario only; counts and narrative are model interpretations, not causal proof.
• Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result.
• The primary result varies across trials; see its full distribution in Behavior diff.

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 854ae1b4e0ace82ea171ae203697d1bb09811012); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}

[github-actions]

Behavior Diff · plugin/skills/behavior-diff/SKILL.md

Run status: Complete — Before 3/3 trials complete · After 3/3 trials complete · behavior extraction complete · report available · runner exit code 0.

• Before — plugin/skills/behavior-diff/SKILL.md at merge-base 7f615f04ed84de645ea4a5896e64424609c54b6e.
• After — plugin/skills/behavior-diff/SKILL.md at PR head 2e49baa409d18cec1c16c6dde22e656297f077dc.
• Scope — only this file differs between Before and After. Every other file stays at the merge-base, so other changes in this PR, including files this instruction references, are not tested.
• Full report — HTML report (single HTML file, no ZIP; requires GitHub sign-in and repository access; expires 2026-10-26T07:01:38Z). Evidence ZIP.
• Workflow run — run 39213095873, attempt 1.

Summary

Answers or actions vary across trials in this scenario.

1. What this edit was meant to change

Model summary of the recorded goals

When trials need extra shared files, the agent should deliver only user-supplied or approved data to both Before and After through a verified stage, never write it into the user's checkout, and stop if staging fails.

A model wrote this short wording from the goals recorded before trials. Only the full goals were recorded before trials; Instruction changes lists each one with its source. Purpose identifies the question, not its answer.

2. What the evidence shows

Based on final answers. An answer describing an action does not prove it happened.

Before

Wrote it to a private /tmp staging folder

1 of 3 trials

Left it unwritten, in chat only

2 of 3 trials

After

Left it unwritten, in chat only

3 of 3 trials

Reported primary result — unchanged

Before: Stopped before launching (3 of 3 trials)

After: Stopped before launching (3 of 3 trials)

Based on reported final answers; these outcomes do not prove execution.

No supported interpretation was supplied; these observations alone do not establish the edit's benefit.

3. What needs your attention

Attention assessment unavailable.

No valid attention assessment is saved for these trials. This does not establish that there are no remaining or additional concerns. The original explanation and trial evidence remain available.

4. What we still cannot tell

• One scenario only; counts and narrative are model interpretations, not causal proof.
• Target assessment unavailable: no valid criterion assessment was recorded. Underlying trial answers and recorded evidence remain available; this is not an assessed-empty result.
• Plain-language narrative unavailable; original extracted comparisons are shown without added explanation.

Comparison references and omitted section links refer to the full report.

---

_{Generated deterministically from this run's result record and Behavior Diff core report (selected core 2e49baa409d18cec1c16c6dde22e656297f077dc); no model wrote this comment. Observed trial behavior is evidence to review, not a pass/fail verdict.}

[kentwelcome]

Post-publication CI observation at head 6406504: Format and Unit passed. The existing dogfood workflow has one failed comparison for plugin/skills/behavior-diff/SKILL.md. Its uploaded result.json reports runner_exit=0, no report, 0/3 complete trials on either side, and extraction unavailable. The artifact does not establish the underlying cause; this is not a successful behavioral comparison or evidence that the new helper failed. The dogfood Action selected core revision 854ae1b. No automatic rerun or production fix was made. Separately requested private live verification is in progress: one single-file case exercises the new helper; a second multi-file case is an explicitly approved historical control, not helper acceptance. Private inputs/reports will not be posted here.

[kentwelcome]

Live-verification follow-up for unchanged head 6406504: both requested private comparisons completed once (3 Before + 3 After + extraction each). The new-helper case used dated Opus 4.5 with high reasoning, delivered a nonempty external shared-data overlay matching frozen originals, and reproduced an unmet After requirement in raw evidence that the automatic report missed. The owner-approved multi-file control used Sol 6.1 high; its prior output-duplication failure did not reproduce, and it is not new-helper acceptance. Actual assistant-event model identities were checked. The PR body now records sanitized results and the unresolved separate dogfood report-generation failure. No product changes or retries followed these runs; all private inputs and reports remain local. PR remains draft.

[kentwelcome]

CI diagnosis follow-up for job 117886131286 (no paid rerun): found a confirmed evidence-collection defect in Action commit 32f1bc3bf8c8a2588a0cca5aca44d9cee4de7b93. scripts/execute.py::_single_run accepts exactly one entry under the run home. I executed the source-extracted collector against synthetic valid reports/grades: one run reports complete with 3/3 per side; adding a second empty nested-run directory reproduces the exact saved failure signature (exit 0, no report, 0/3 both sides, extraction unavailable), while the outer report still exists. Nested comparisons are a reachable explanation when dogfooding the Behavior Diff skill: trial subprocesses inherit BEHAVIOR_DIFF_HOME and may invoke the runner. This is a strong candidate for this job, not proof that nesting occurred: runner stdout/stderr and private run trees were discarded, so the uploaded result cannot distinguish ambiguous run selection from absent output. The 0/3 counts are collector defaults, not proof that no trials executed. Also verified core revision 854ae1b is the merge of PR head 6406504 into base 7f615f0, not an old bundled core. Recommended correction: collect the explicitly identified outer run, and preserve bounded sanitized collection diagnostics before cleanup. No repository changes or workflow retry performed.

[kentwelcome]

Diagnosis cross-check: fetched behavior-diff.sh from the exact CI core revision 854ae1b and confirmed it is byte-identical to the runner inspected locally. Also exercised the same source-extracted collector with an empty run home: it produces the identical failure signature. Therefore the saved result is consistent with ambiguous nested runs OR missing output and does not identify which occurred. Nested execution remains a reachable hypothesis, not a confirmed event. No model calls or code edits were made.

[kentwelcome]

Correction review round (head 6406504 plus unpublished local fix): Spec reviewer APPROVE. Standards reviewer REVISE for an existing PR1 edge case: prepare_shared.py treats BEHAVIOR_DIFF_HOME="" as cwd rather than using the default home, unlike the runner/hooks. Both reviewers accept trial-local storage isolation and opt-in real nested-run regressions. The empty-home fallback will be corrected and regression-tested before publication; full re-review and corrected-head dogfood success remain required. Nested smoke failed before (3 root runs rather than1 for both hosts), then all30 manual/boundary checks passed with isolation. No merge yet.

[kentwelcome]

Final correction review: independent Standards and Spec reviewers both APPROVE head 2e49baa against base 7f615f0. All material findings resolved: trial-local homes prevent nested-run contamination; empty home overrides retain default hook-baseline semantics. Both regression paths demonstrated failing-before/passing-after. All31 boundary/manual smoke tests passed, including actual nested shipped runners through fake Claude/Codex; required deterministic suites, formatting, syntax and whitespace checks passed. Scope audit: no Action API, report redesign, generator or multi-file feature added. Fresh dogfood will use this corrected branch/core revision and the failed SKILL.md selection. Original incident cause remains unproven because prior logs were deleted. No merge until fresh collection/publication and CI pass.

[kentwelcome]

Merge gate satisfied for head 2e49baa (base 7f615f0): both independent read-only reviewers APPROVE; final31 boundary/manual smoke checks and required suites passed; Format and Unit passed. Fresh dogfood run https://github.com/spacedock-dev/behavior-diff/actions/runs/39213095873 used this exact corrected core/head and succeeded through publication. Inspected uploaded result.json: complete, Before3/3, After3/3, extraction complete, runner_exit0, error null, report_available true. No unresolved material review findings. The original historical nesting cause remains unproven, but the reproduced isolation defect is fixed and the current CI report path is verified. Merging under the owner request.