[ting-hong-shieh]

What

Add a DeepSWE v1.1 evaluation plan that compares both stage-router pickers at 0.5 with fixed efficient and capable baselines. The plan generates four configs from the published qualification profile, keeps the agent-facing route ID fixed, and records config hashes and the Switchyard revision.

It defines a 40-attempt pilot, shared task selection, result accounting, and cost estimation before larger runs. A live comparison is needed because fixed-trajectory replay cannot measure the task-success or cost effects of changed model outputs.

Why

Follow up on the clean-production scorer change in #984 and the reproducible-evaluation direction in #1026. Related: #264. This adds an evaluation procedure; it makes no new quality or cost claim.

Notes for reviewers

Start with benchmark/STAGE_ROUTER_EVALUATION.md. The benchmark README links to the plan. Existing qualification profiles remain the source for model, provider, window, hold, and handoff settings.

Validation on main 9ddd4879:

• Executed the documented config-generation block; all four outputs parse as TOML.
• Loaded all four configs with switchyard-server --dry-run using a non-secret placeholder key; all passed without provider calls.
• Strict MkDocs build and git diff --check: passed.

No live benchmark or paid model call was run. This draft is for review of the procedure and measurement contract before spending on an evaluation.