feat: explain change assumptions in an evidence-backed Summary

.agents/skills/run-behavior-diff-human-evaluation/references/workflow.md

@@ -187,18 +187,18 @@ Do not represent an answer describing an action as evidence of execution.
 ## Quiz and scoring
 
 `build` creates blinded excerpts from the original saved report HTML. It retains
-the generated headline, case-evidence context, Before/After cards and supported
-starts, practical meaning, attention findings, target gaps, and visible limits.
-It removes instruction intent/diff, full scenario/expected behavior, commit
-metadata, source-intent relationship labels, and links to unblinded evidence.
-Case context is preserved separately from the removed purpose; extraction must
-not use it to restate the instruction. An exact repetition of the Summary intent,
-any full recorded goal, or the full supplied expectation or inferred aim from
-Instruction changes is rejected. This guard cannot detect semantic paraphrases.
-The projection never rewrites prose or synthesizes missing summaries. The v3
-projection accepts only the current Summary structure and fails closed on older
-layouts. Already-built sessions keep their frozen bytes; do not rebuild an
-active evaluation to adopt a new presentation.
+the bounded observation section: case-evidence context, Before/After cards and
+supported starts, practical meaning, attention findings, target gaps, and limits.
+The assessment-derived headline is replaced by a neutral observation heading,
+not another finding. Expected criteria, method/rationale, next actions, instruction
+intent/diff, commit metadata, source-intent relationship labels and links to
+unblinded evidence are excluded because they can reveal the keyed edit.
+Case context must not restate the instruction. Exact repetition of the Summary
+intent, any full goal, supplied expectation or inferred aim is rejected; this
+guard cannot detect semantic paraphrases. Observation prose is never rewritten.
+The v4 projection accepts only the current five-step Summary structure and fails
+closed on older layouts. Already-built sessions keep their frozen bytes; do not
+rebuild an active evaluation to adopt a new presentation.
 
 `serve SESSION --port 0` prints an available loopback URL. Only public quiz assets
 are served before submission. The answer key stays on the server. Check desktop

.agents/skills/run-behavior-diff-human-evaluation/scripts/quiz.py

@@ -464,25 +464,11 @@ def _claim_shape(claim):
     _shape(claim, [("strong", None), ("p", None)])
 
 
-def _intent_case_context(body, source_intents):
-    """Reject case context that repeats displayed or full-detail source intent."""
-    children = _children(body)
-    if (
-        len(children) not in (2, 3)
-        or children[0].tag != "div"
-        or not children[0].has_class("intent-heading")
-        or children[1].tag != "p"
-        or not children[1].has_class("story-intent")
-    ):
-        raise ValueError("Unknown intended-change step.")
-    if len(children) == 2:
-        return None
-    context = children[2]
-    if context.tag != "div" or not context.has_class("summary-case-context"):
-        raise ValueError("Unknown intended-change step.")
+def _case_context(context, source_intents):
+    """Reject observed context that repeats a displayed source goal."""
     _claim_shape(context)
     shown = " ".join(_plain(context).split())
-    for intent in (children[1], *source_intents):
+    for intent in source_intents:
         text = " ".join(_plain(intent).split())
         if intent.has_class("edit-aim"):
             for prefix in ("Supplied expectation: ", "Likely aim: "):
@@ -491,7 +477,6 @@ def _intent_case_context(body, source_intents):
                     break
         if text and text in shown:
             raise ValueError("Case context repeats the intended change.")
-    return context
 
 
 def _blinded_report(source):
@@ -532,6 +517,11 @@ def _blinded_report(source):
         raise ValueError("Unknown Summary disclosure order.")
     _shape(details[1], [("summary", None), ("ul", "evidence-limits")])
     story_children = _children(story)
+    # A leading status badge states the verdict; it is validated, never projected.
+    if story_children and story_children[0].has_class("summary-status"):
+        if story_children[0].tag != "p":
+            raise ValueError("Unknown summary status.")
+        story_children = story_children[1:]
     if (
         len(story_children) != 2
         or story_children[0].tag != "h2"
@@ -541,44 +531,47 @@ def _blinded_report(source):
     ):
         raise ValueError("Unknown summary story structure.")
     steps = _children(story_children[1])
-    if (
-        len(steps) != 4
-        or any(node.tag != "li" or not node.has_class("story-step") for node in steps)
-        or not steps[2].has_class("story-attention")
-        or not steps[3].has_class("story-limits")
+    names = ("finding", "expectation", "method", "evidence", "next")
+    if len(steps) != len(names) or any(
+        step.tag != "li"
+        or not step.has_class("story-step")
+        or not step.has_class(f"story-{name}")
+        for step, name in zip(steps, names)
     ):
         raise ValueError("Unknown story steps.")
     bodies = [
         _shape(step, [("span", "story-number"), ("div", "story-body")])[1]
         for step in steps
     ]
-    source_intents = [
-        node
-        for node in nodes
-        if node.has_class("purpose-goal-text") or node.has_class("edit-aim")
-    ]
-    case_context = _intent_case_context(bodies[0], source_intents)
+    # Check digest and qualification summarize the assessment; drop them but
+    # accept only these exact optional siblings before the observations.
     evidence_shape = [("h3", None)]
-    if any(node.has_class("summary-context") for node in _children(bodies[1])):
-        evidence_shape.append(("p", "summary-context"))
-    evidence_shape.extend([("p", "summary-provenance"), ("div", "summary-pair")])
-    if any(node.has_class("primary-result-context") for node in _children(bodies[1])):
-        evidence_shape.append(("section", "primary-result-context"))
-    if any(
-        node.has_class("summary-interpretation-unavailable")
-        for node in _children(bodies[1])
+    for tag, optional in (
+        ("div", "summary-check-digest"),
+        ("p", "summary-qualification"),
     ):
-        evidence_shape.append(("p", "summary-interpretation-unavailable"))
-    else:
-        evidence_shape.append(("div", "summary-why"))
-    evidence_shape.append(("nav", "evidence-nav"))
-    evidence_body = _shape(bodies[1], evidence_shape)
-    if _plain(evidence_body[0]) != "What the evidence shows":
-        raise ValueError("Unknown evidence heading.")
-    pair = _single(
-        [node for node in evidence_body if node.has_class("summary-pair")],
-        "summary pair",
+        if any(node.has_class(optional) for node in _children(bodies[3])):
+            evidence_shape.append((tag, optional))
+    evidence_shape.extend([("div", "summary-observations"), ("nav", "evidence-nav")])
+    evidence_parts = _shape(bodies[3], evidence_shape)
+    evidence_heading, observations = evidence_parts[0], evidence_parts[-2]
+    children = _children(observations)
+    shape = [("p", "summary-provenance"), ("div", "summary-pair")]
+    if any(node.has_class("primary-result-context") for node in children):
+        shape.append(("section", "primary-result-context"))
+    shape.append(
+        ("p", "summary-interpretation-unavailable")
+        if any(
+            node.has_class("summary-interpretation-unavailable") for node in children
+        )
+        else ("div", "summary-why")
     )
+    for optional in ("summary-case-context", "summary-caution"):
+        if any(node.has_class(optional) for node in children):
+            shape.append(("div", optional))
+    shape.append(("div", "summary-attention"))
+    _shape(observations, shape)
+    pair = children[1]
     cards = _shape(
         pair,
         [
@@ -588,9 +581,9 @@ def _blinded_report(source):
         ],
     )
     for card in (cards[0], cards[2]):
-        children = _children(card)
-        start = bool(len(children) > 1 and children[1].has_class("summary-start"))
-        empty = bool(children and children[-1].has_class("summary-empty"))
+        parts = _children(card)
+        start = bool(len(parts) > 1 and parts[1].has_class("summary-start"))
+        empty = bool(parts and parts[-1].has_class("summary-empty"))
         card_shape = [("h4", "summary-side-label")]
         if start:
             card_shape.append(("div", "summary-start"))
@@ -603,28 +596,52 @@ def _blinded_report(source):
         _shape(card, card_shape)
         if start:
             start_shape = [("p", "summary-start-text"), ("p", "summary-start-source")]
-            if len(_children(children[1])) == 3:
+            if len(_children(parts[1])) == 3:
                 start_shape.append(("span", "summary-start-arrow"))
-            _shape(children[1], start_shape)
-    if evidence_body[-2].tag == "div":
-        _claim_shape(evidence_body[-2])
-    _attention_shape(bodies[2])
-    limits_shape = [("h3", None)]
-    if any(node.has_class("summary-caution") for node in _children(bodies[3])):
-        limits_shape.append(("div", "summary-caution"))
-    limits_shape.append(("ul", "summary-notices"))
-    limits = _shape(bodies[3], limits_shape)
-    if len(limits) == 3:
-        _claim_shape(limits[1])
-    # Remove the intended change, its source label and every navigation link.
-    # Case context is observation context; the reader sees it beside the evidence.
-    if case_context is not None:
-        evidence = bodies[1].children
-        evidence.insert(evidence.index(evidence_body[0]) + 1, case_context)
+            _shape(parts[1], start_shape)
+    source_intents = [
+        node
+        for node in nodes
+        if any(
+            node.has_class(name)
+            for name in ("purpose-goal-text", "edit-aim", "story-intent")
+        )
+    ]
+    for node in children:
+        if node.has_class("summary-case-context"):
+            _case_context(node, source_intents)
+        elif node.has_class("summary-why") or node.has_class("summary-caution"):
+            _claim_shape(node)
+    _attention_shape(children[-1])
+    # Keep only the bounded observation section. Assessment verdicts, expected
+    # criteria, preparation rationale and next actions can reveal the keyed edit.
     trimmed_story = _Node(
         story.tag,
         story.attrs.items(),
-        [story_children[0], _Node("ol", story_children[1].attrs.items(), steps[1:])],
+        [
+            _Node(
+                "h2",
+                [("class", "summary-headline")],
+                ["Observed Before / After behavior"],
+            ),
+            _Node(
+                "ol",
+                story_children[1].attrs.items(),
+                [
+                    _Node(
+                        "li",
+                        steps[3].attrs.items(),
+                        [
+                            _Node(
+                                "div",
+                                [("class", "story-body")],
+                                [evidence_heading, observations],
+                            )
+                        ],
+                    )
+                ],
+            ),
+        ],
     )
     fragment = "".join(
         _safe_fragment(node) for node in (trimmed_story, direct[1], details[1])
@@ -774,7 +791,7 @@ def build_quiz(session: Path, repo_root: Path) -> None:
         "reports": reports,
         "answer_key_sha256": _hash((session / "answer-key.json").read_bytes()),
         "session_sha256": _hash((session / "session.json").read_bytes()),
-        "summary_method": "saved-html-summary-panel-v3",
+        "summary_method": "saved-html-summary-panel-v4",
     }
     _atomic_file(session / "quiz-build.json", _bytes(receipt))
 

README.md

@@ -210,6 +210,35 @@ without an assessment. No output does not mean stuck. When local process identit
 cannot be checked, status stays explicitly uncertain. Older runs without progress
 metadata still have their reports.
 
+### Explain the scenario and freeze its checks
+
+The skill prepares a reviewed scenario record alongside purpose before launching
+trials. It describes the task, why the selected data tests the change assumption,
+the necessary conditions, the limits, and separate observable checks for each goal.
+
+```bash
+behavior-diff.sh --agent claude --file SKILL.md --task "<task>" \
+  --purpose-file <private-purpose.json> --scenario-file <private-scenario.json>
+```
+
+`--scenario-file` requires purpose. The runner validates both before any model call
+and excludes them from both trial copies. Only the sanitized scenario record goes
+to assessment and the report; the private preparation notes do not. See the
+[record contract](plugin/skills/behavior-diff/references/scenario-and-test-data.md)
+for its fields and privacy boundary.
+
+The assessor must evaluate every frozen check without changing its definition.
+Reading a PR description and checking the implementation are separate checks;
+one cannot stand in for the other. Each source-consistency check also names its
+required recorded sources. These sources get priority within the existing evidence
+budget; an unrelated readable file cannot substitute for them.
+
+Invalid evidence rejects only the affected check, which stays **Not assessed** with
+its reason. Other validated findings remain visible. Missing, duplicate or rewritten
+check definitions still reject the whole assessment. Partial coverage never becomes
+an all-pass result. Runs without a scenario record show missing setup information
+explicitly and identify any criteria written by the assessor after the trials.
+
 ### Report wording
 
 Behavior Diff applies installed Humanizer prose guidance while generating
@@ -298,25 +327,21 @@ save the complete environment, event payload, or credential-bearing clone URL.
 A **run** compares the instruction versions. A **trial** is one agent execution on
 one side. Each run creates a local HTML report with six tabs:
 
-- **Summary** tells a four-part numbered story: the intended change, what the
-  evidence shows, what needs your attention, and what this means. Illustrated
-  Before/After cards retain every observed branch and its trial count, and
-  distinguish plans, answers, and recorded actions. The primary result stays
-  visible when a secondary comparison leads the story.
-  **What needs your attention** explains relevant tradeoffs with pictures,
-  consequences, applicability, and possible actions. An expected tradeoff can
-  matter too. An assessed-empty result is distinct from an unavailable
-  assessment; neither hides the underlying comparisons.
-  **See why this matters** opens the detailed attention explanation.
-  **Full scenario and expected behavior** contains the setup and nested full
-  prompt; complete evidence limits remain in their own disclosure.
-- **Understand the change** preserves the original explanation. Attention detail
-  adds reasoning, all branch counts with links to supporting trials, cited
-  context, and uncertainty instead of repeating the Summary pictures and actions.
-  Count-only saved evidence explicitly marks trial attribution unavailable.
-  Optional **Check the evidence** disclosures link to raw comparisons.
-  **Not relevant here** hides a Summary finding's body for this report session;
-  **Show again** or reloading restores it. This does not change the evidence.
+- **Summary** follows one five-step reading sequence: what we found; expected
+  versus observed behavior; how we tested it; supporting evidence; and next action.
+  Validated checks distinguish unmet After requirements, an already-correct
+  baseline, preservation, improvement in these trials, and inconclusive evidence.
+  The same mistake on both sides can still leave an After requirement unmet.
+  Missing or rejected assessments never become a no-gap result. Recorded task,
+  model, trial counts and evidence limits explain the test; private preparation
+  rationale is not invented when it was not saved in report inputs.
+  Repository/PR links, compared revisions and model remain visible in a compact
+  header. Full task, reading help and detailed limits are expandable.
+- **Understand the change** explains the same assessment conclusion. Original
+  model interpretation is labeled separately, not promoted to a competing verdict.
+  Detailed checks link their recorded sources, output excerpts and trial outcomes.
+  Attention findings retain consequences, applicability and uncertainty; **Not
+  relevant here** hides a finding's body for this report session only.
 - **Instruction changes** shows the saved aim, filename, line counts, and
   complete instruction diff. **Likely aim** marks an inferred interpretation;
   **Supplied expectation** marks an explicit expectation. Missing explanations
@@ -435,12 +460,20 @@ Expected behavior is labeled as supplied, or explicitly unavailable. The evidenc
 limits state trial counts, provenance, and grading limits. Self-reported actions
 remain distinct from captured tool calls.
 
-The structured `report-data.json` uses schema version 16, including per-group
-`decisions.trial_summaries`, visual `summary`, sourced instruction `intent`,
+The structured `report-data.json` uses schema version 19, including per-group
+`decisions.trial_summaries`, assessment-derived `summary.state`, `summary.meaning`,
+`summary.checks` and `summary.next_action`, sourced instruction `intent`,
 saved `content.task`, optional narrative, nullable `metadata.source` and
-`metadata.ci_context`, and `edit_hunks`. The optional
+`metadata.ci_context`, and `edit_hunks`. `decisions.target_rejection` explains
+a rejected whole assessment without dropping underlying evidence. Individual
+`target_assessment.criteria[].rejection` values retain check-local diagnostics,
+with empty outcome arrays rather than invented uncertain results. The optional
 `decisions.purpose_summary` gives the Summary a short, labeled paraphrase of
 the frozen goals; Instruction changes retains their complete text and provenance.
+`decisions.scenario_record` carries the validated setup, frozen checks and required
+source groups from run configuration. The Summary explains the test rationale and
+conditions even when the automatic assessment is unavailable. These expectations
+are not trial evidence.
 The task comes from the run's `task.md`, or the capsule's copy
 when absent from the run. A scenario description never replaces missing task text.
 Regenerate reports from their original run artifacts;
@@ -449,10 +482,10 @@ capsule directory, recorded model label, and original configuration. Rendering
 does not run trials or call a model.
 
 Old `decisions.json` observations can be re-rendered with the new layout.
-Without narrative, the Summary uses original choice text and marks explanations
-unavailable. Missing trial summaries, aim interpretations, and edit mappings
-remain unavailable. Trial summaries are produced in the existing extraction
-call; rendering never substitutes an overall conclusion for a trial's summary.
+Without validated target checks, the Summary states that assessment is unavailable;
+observed comparisons remain accessible. Missing narrative, trial summaries, aim
+interpretations, and edit mappings remain unavailable. Trial summaries are produced
+in the existing extraction call; rendering never invents a trial's summary.
 To obtain them, explicitly rerun extraction on saved evidence after approving
 its model cost; no new trials are needed. Rendering never triggers extraction.
 Without a valid purpose summary, a single short recorded goal can appear verbatim.