Repository navigation
docs: add preservation-aware harness evaluation - #17
Merged
Merged
Conversation
Reflect arXiv:2611.07624: harness changes can raise average scores while losing previously passing tasks, increasing cost and failing to generalize. Add a fixed V-suite before/after procedure that reports new successes, preserved tasks, regressions, blockers and cost; reject changes that regress must-preserve tasks. Add decision #25 and explicitly keep the process manual rather than building an automatic harness optimizer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reflect the preservation problem described in Beneath the Surface (arXiv:2611.07624): harness changes can improve average scores while losing previously passing tasks, raising cost and generalizing unevenly.
docs/harness-evaluation.md: fixed V-suite baseline, same-condition rerun, per-task new/preserved/regressed/blocked/unverified reporting, cost, adoption rule and generalization limit.docs/validation.mdand update the decision count to 25.Current limitation: the 7/7 mechanism checks and the six-task UI pilot are not a harness-wide V-suite baseline. The procedure is defined; the fixed suite and before/after runs still need to be built.
Verification: offline 1/1;
git diff --checkclean.