Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ jobs:
plugin/skills/behavior-diff/scripts/status.py \
plugin/skills/behavior-diff/scripts/source.py \
plugin/skills/behavior-diff/scripts/ci_context.py \
plugin/skills/behavior-diff/scripts/prepare_shared.py \
plugin/skills/behavior-diff/scripts/reporting/*.py

- name: Run hook unit checks
Expand All @@ -103,6 +104,9 @@ jobs:
python3 tests/ci-context-test.py
python3 tests/report-ci-context-test.py

- name: Check shared-data preparation boundaries
run: python3 tests/shared-data-test.py

- name: Run decision unit check
run: python3 plugin/skills/behavior-diff/scripts/decisions.py --check

Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ For the full deterministic suite, run:
```bash
bash tests/hooks-test.sh
python3 plugin/skills/behavior-diff/scripts/decisions.py --check
python3 tests/shared-data-test.py
bash tests/live-report-contract.sh
python3 tests/source-provenance-test.py
python3 tests/report-source-test.py
Expand Down
2 changes: 2 additions & 0 deletions CODING_GUIDELINES.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,10 +190,12 @@ python3 -m py_compile \
.agents/skills/release-behavior-diff/scripts/bump-version.py \
plugin/skills/behavior-diff/scripts/codex_model.py \
plugin/skills/behavior-diff/scripts/decisions.py \
plugin/skills/behavior-diff/scripts/prepare_shared.py \
plugin/skills/behavior-diff/scripts/render.py \
plugin/skills/behavior-diff/scripts/reporting/*.py
bash tests/hooks-test.sh
python3 plugin/skills/behavior-diff/scripts/decisions.py --check
python3 tests/shared-data-test.py
bash tests/live-report-contract.sh
bash tests/release-workflow-test.sh
python3 .agents/skills/release-behavior-diff/scripts/bump-version.py --check
Expand Down
36 changes: 33 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,34 @@ In Claude Code, you can also run:
```

Behavior Diff finds the changed instruction file and uses your request as the
comparison task. Once the task is known, it runs the comparison and opens the
report.
comparison task. Once the task is known, and any shared trial data is verified,
it runs the comparison and opens the report.

### Shared trial data

Some tasks need files that your project does not contain, such as sample
inputs. Git-based trials start from your last commit, so untracked or
uncommitted files in your checkout never reach them. Behavior Diff does not
generate such data. When you supply or approve the files, the skill can deliver
them to every trial:

1. A private helper, `prepare_shared.py`, copies your project base, the reviewed
data directory, and the selected Before and After instructions into a new
private stage outside your project. Your checkout is not changed.
2. The helper records a freeze of the stage. It checks the freeze again
immediately before launch, and it stops when a file changed.
3. The skill runs the normal runner from the stage. Both sides receive the same
data, and only the selected instruction differs.

The helper accepts one instruction file and plain files only. It rejects
symbolic links, special files, submodules, unsafe paths, inputs over
`BEHAVIOR_DIFF_COPY_CAP_KB` (default 200 MiB), and Git attributes that change
archived files. Purpose files, source metadata, and evaluator notes stay out of
the stage. The helper does not scan data for secrets, so review every shared
file first. The freeze detects changes, but it is not a sandbox. Reports for a
staged run stay in the stage under `home/runs/`; keep the stage after the run.
The [skill reference](plugin/skills/behavior-diff/references/scenario-and-test-data.md)
contains the complete commands and limits.

### Trial and extraction models

Expand Down Expand Up @@ -438,14 +464,18 @@ change.

Behavior Diff runs each side in a separate copy of the project. The after side
contains only the instruction change that you selected. Other uncommitted files
do not enter the experiment.
do not enter the experiment. Reviewed shared trial data, when used, enters both
sides identically.

Reports stay on your machine under:

```text
${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/runs/
```

A run with [shared trial data](#shared-trial-data) keeps its reports in its
private stage under `home/runs/`.

A report can quote code and agent output from your project. Review the report
before you share it.

Expand Down
78 changes: 76 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ scripts handle execution, evidence, and reporting.
+----------------------------+----------------------------+
v
+---------------------------------------------------------+
| Conditional shared-data delivery |
| Only when the task needs files the project base lacks: |
| prepare_shared.py freezes project, Before, and home; |
| verify passes immediately before launch |
| Otherwise the runner starts from the user's project |
+----------------------------+----------------------------+
v
+---------------------------------------------------------+
| Trial execution |
| behavior-diff.sh builds independent Before/After copies |
| run-trial.sh runs the same task/model in each copy |
Expand Down Expand Up @@ -111,7 +119,11 @@ remain the only source of that guidance.

The skill selects the trial host and model, calls the runner, and explains the
result. It owns scenario quality and interpretation, not filesystem copying or
report formatting.
report formatting. When the task needs common files that the project base
lacks, the skill reviews files that the user supplied or approved and delivers
them through a frozen stage (see
[Shared-data delivery](#shared-data-delivery)). The skill does not generate the
scenario or the data.

Before execution, the skill records distinct concise goals and their provenance
(`session`, `commit`, or `diff`; `explicit` or `inferred`) in a reviewed purpose
Expand Down Expand Up @@ -227,6 +239,12 @@ instruction versions without asking a model to rewrite either version.
the shared `task.md`, and the selected model. The runner starts all trials
as background processes and waits before analysis.

`run-trial.sh` gives every host a trial-local `BEHAVIOR_DIFF_HOME` at
`<trial>/behavior-diff-home`, outside `project/`. Nested comparisons and hook
baselines cannot populate the orchestrator's run directory or another trial's
store through the inherited variable. This applies to Claude Code, Codex, Pi,
and OMP; it isolates storage defaults, not filesystem access.

Each side contains `N` independent copies, not one shared directory:

```text
Expand Down Expand Up @@ -260,7 +278,58 @@ The configuration records the target path, scenario, Before/After labels, and
validated `purpose`. `--purpose-file` is frozen before trials and excluded from
trial copies, including when supplied from within the source project.
Uncommitted setup files do not enter Git-based trials. Required task state must
already exist in the base project or in a deliberately prepared fixture.
already exist in the base project, in a deliberately prepared fixture, or in a
shared-data stage.

### Shared-data delivery

The private helper
[`prepare_shared.py`](../plugin/skills/behavior-diff/scripts/prepare_shared.py)
delivers a reviewed data directory to every trial without changing the user's
checkout or adding runner options. It does not generate data, launch models,
or infer provenance.

```text
Project base (Git HEAD or folder copy) --+
Reviewed data directory -----------------+--> stage/project: scratch commit
Selected Before (HEAD, file, or absent) -+ + After in the working tree
minus --exclude paths stage/before: present Before
stage/home: ABSENT marker
stage/freeze.json
|
verify, then behavior-diff.sh from stage/project
with BEHAVIOR_DIFF_HOME=stage/home
and --before-file stage/before when present
```

`prepare` resolves Before in the runner's order: an explicit file, the target
in Git HEAD, then the newest hook baseline. Only a store entry can mean
`ABSENT`. It copies the base, the data, and the selected Before into a new
scratch Git repository and commits them. The repository has no remote, and its
hooks and signing are off. The helper then writes After into the working tree.
A present Before, from HEAD or a file, is also frozen as the stage's `before`
file outside the scratch project. The skill passes that file to the runner
with `--before-file`, so the report labels Before with the frozen file rather
than a scratch commit. An absent Before has no such file; it receives a marker
in the stage's `home/baselines/`, and the runner resolves it there. The
existing runner then builds every trial from the scratch commit, unchanged.

The helper rejects links, special files, Git internals, submodules, unsafe or
colliding paths, data that replaces the instruction or an excluded path, and
inputs over `BEHAVIOR_DIFF_COPY_CAP_KB`. The caller excludes in-project purpose,
source, and evaluator files with `--exclude`. `prepare` also requires
`git archive` of the scratch commit to match the expected files. Therefore,
`.gitattributes` rules such as `export-ignore`, `export-subst`, or filters that
change archived files fail before launch.

`freeze.json` records the original project and commit, the target, the Before
mode, the scratch commit, exclusions, and the content and executable bit of
every archived and working file. `verify` compares the stage, including the
frozen Before file or absent marker, with this record immediately before
launch. The freeze detects changes. It is not a sandbox, and the stage files
are not read-only. Source provenance still names the original repository and
revisions, never scratch commits. Runs and reports stay under the stage's
`home/runs/`.

### Trial execution sequence

Expand Down Expand Up @@ -742,6 +811,11 @@ State lives under `${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}/`:
- `human-evaluations/`: private source snapshots, frozen scenarios/questions,
original reports, quiz projections, and human submissions for manual evaluations.

A shared-data run uses its own stage instead: `project/` (the scratch base),
`before` (a present Before version), `home/` (its `BEHAVIOR_DIFF_HOME`, with
any `ABSENT` marker and its runs), and `freeze.json`. The stage stays after the
run because it holds the reports.

The runner prepares copies without changing the source project. Separate
working directories are not a universal security sandbox. Tool restrictions
depend on the host, and permitted shell commands can have side effects.
Expand Down
Loading
Loading