Repository navigation
fix(submission): refuse a decision restating a pinned source's flow count - #83
Merged
Merged
Conversation
The planner instruction asked for a decision on every measure, so the model wrote its own gross/net decision beside the "How the flow is counted" decision propose_plan records for each pinned source, and the plan listed the flow twice. The instruction now asks for a decision on a measure no pinned source settles, and says a pinned source fixes its flow count, with another count reached by pinning that count's spec_id.
…ount Under the planner instruction that leaves a pin's flow count to propose_plan, 14 of 131 pinned plans still listed the flow twice: the model wrote its own decision fixing the measure beside the code's "How the flow is counted" decision for the same pin. propose_plan now refuses such a draft as measure_fixed_by_pin when a requirement citing it also cites the code's decision for its own pin.
# Conflicts: # services/harness/tests/context_baseline.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #78, whose planner instruction this PR builds on; merge #78 first.
Under #78's instruction, 14 to 19 of about 130 pinned plans per paired planner-entry eval run still listed the flow choice twice: the model wrote its own decision, such as "Whether to measure ODA flows as gross or net -> Gross (default)", beside the "How the flow is counted" decision
propose_planrecords for the same pinned source (submission/service.py_measure_decisions). Across 616 such decisions in older runs, 606 restate the flow; the other 10, from malaria and health tasks, choose between two pinned readings, which the instruction asks the model to state as a source decision naming the candidates.propose_plannow refuses that draft with a new problem,measure_fixed_by_pin(planner/tool_results.py)._measure_fixed_by_pin, the last check inadmit'sblockedchain, refuses a decision among the call's own drafts whosefixeslistsmeasurewhen a dataset requirement citing it also cites a different decision thatis_search_pinmarks for that requirement's own pin. That decision comes from the decisions this call settles plus the carriedprevious.decisionsit neither removes nor restates (_decisions_after). The refusal's detail names the code's decision and its chosen count, and its message tells the model that the pin decides the measure and that a choice between candidate readings is stated as the source decision. The check runs at the door, never in aPlanvalidator, so a stored plan holding such a decision still loads and can be amended. On an amendment, a carried requirement the code writes nothing for this call, and a draft restating the code's ownsource_<pin>slot, are recorded as before, so the refusal anduser_decision_contradictednever point the model opposite ways.No new
FixedFieldorDecisionFixesvalue is added, which keeps the stored plan shape and its hash unchanged, and nothing is dropped from the display, where hiding a cited decision would also hide a "net" choice beside a gross pin. The instruction andtests/context_baseline.jsonare unchanged.The paired eval ran 70 tasks three times per arm, both arms at the same time, with #78's head as the base. Pinned plans listing the flow twice fell from 19 of 129 to 3 of 131, a difference of -12.4 points (95% interval -20.7 to -4.9). Final acceptance showed no detected change, 192 to 194 of 210 (+1.0 points, 0.0 to +2.4), and
propose_plancalls to acceptance went from 1.44 to 1.46 (+0.02, -0.06 to +0.11). First-proposal acceptance showed no detected change either. The refusal fired in 22 runs, and all 22 ended as their base runs did. The three duplicates left carry nofixes, as in the base, and none followed a refusal.A re-pin on an amendment whose requirement still cites a carried decision fixing the measure lists the flow twice with no refusal; this PR checks only the call's own drafts. The added line in
admittouches the function #80 and #82 edit, so whichever merges later needs a small rebase.Testing: new tests cover a restated flow decision refused with its full detail; an unpinned requirement, a fix on another field and a pin offering no measure, each recorded; a stored plan holding the decision loading and amending; and three amendment rows (the code's own slot restated, a pin settled after its requirement was recorded, and that case with a silent user). The refusing row and the three amendment rows fail without the change. The full harness suite passes (7171 passed, 1 skipped, 1 xfailed). ruff and ty are clean.