Repository navigation
docs: record a real scode-shepherd run as a future eval case - #510
Merged
Merged
Conversation
scode
force-pushed
the
shepherd-testcase-patreon-tags
branch
from
November 7, 2026 05:24
b25b3ac to
e4608be
Compare
Until 2026-11-06 scode-shepherd had only run on throwaway tasks. That day it carried a real goal through to four approved PRs: letting patreon-buddy, a Rust app, follow a Patreon creator's tags alongside collections, which meant changing how the app identifies a subscription in its storage, sync, and UI layers. The run makes a good eval case: hard enough that the cheap worker model gave up twice, yet finished in about two hours for about $12 at API prices. Nobody will remember the details by the time an eval gets written, so this records what one would need to replay it: the starting commit, the skill version, the ask, and the outcome to compare against. No eval is written yet. The ask is condensed from the original conversation, which wandered through an assessment, a detour about getting an auth token to the agent, and the planning questions, into the three prompts to give and the answers to choose. The record goes in a new testcases/ directory that only the maintainer README points to. An agent running the skill reads SKILL.md and the files it names, so it never sees an expected outcome that could steer the run being measured.
scode
force-pushed
the
shepherd-testcase-patreon-tags
branch
from
November 7, 2026 16:28
e4608be to
ba55b09
Compare
scode
marked this pull request as ready for review
November 8, 2026 17:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Until 2026-11-06 scode-shepherd had only run on throwaway tasks. That day it carried a real goal through to four
approved PRs: letting patreon-buddy, a Rust app, follow a Patreon creator's tags alongside collections, which meant
changing how the app identifies a subscription in its storage, sync, and UI layers. The run makes a good eval case:
hard enough that the cheap worker model gave up twice, yet finished in about two hours for about $12 at API prices.
Nobody will remember the details by the time an eval gets written, so this records what one would need to replay it:
the starting commit, the skill version, the ask, and the outcome to compare against. No eval is written yet.
The ask is condensed from the original conversation, which wandered through an assessment, a detour about getting an
auth token to the agent, and the planning questions, into the three prompts to give and the answers to choose.
The record goes in a new testcases/ directory that only the maintainer README points to. An agent running the skill
reads SKILL.md and the files it names, so it never sees an expected outcome that could steer the run being measured.