Skip to content

fix(ranking): leave out primary metrics whose baselines are within noise; dashboard footnote (v0.31.1) - #150

Merged
alperkent-cmi merged 1 commit into
mainfrom
fix/ranking-anchor-noise
Nov 2, 2026
Merged

alperkent-cmi merged 1 commit into
mainfrom
fix/ranking-anchor-noise

Conversation

@alperkent-cmi

Copy link
Copy Markdown
Collaborator

Requested by Alp · project thread

What (v0.31.1)

Two follow-ups to the anchored ranking (v0.30.0), stacked on the model-cards PR (v0.31.0).

  • Baselines within noise. Anchoring divides by the gap between the two baselines. When that gap is about the size of its noise, the anchored scores blow up (on the integration fixture the fairness interval reached -47). With the ranking bootstrap on, a primary metric is now anchored only when the one-sided 95% lower bound of the baseline gap is above 0. Otherwise it is left out with a warning, listed in the report, and the scores and intervals are recomputed without it. On the fixture this drops demographic_parity_gap_error, whose gap was above 0 in only 13% of draws. The reasoning is in docs/verification.md (Baseline-anchored scores).
  • Dashboard footnote now describes the anchored scores and bootstrap ties instead of the Welch t-test.
  • Report rebuilt by synthdata-plot keeps the anchored primary table and the left-out list; primary_metrics.csv gains a dropped column.

Checks

uv run ty check, ruff check/format, unit tests (896 passed), CPU integration tier (91 passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_013mqpy2RQu95wcEnosoxLFj

…ise; dashboard footnote; v0.31.1

- Anchoring divides by the baselines' gap; when that gap is within its
  own noise the scores blow up (fixture: fairness interval down to -47).
  With the ranking bootstrap on, a metric is anchored only when the
  one-sided 95% lower bound of the baseline gap is above 0; otherwise it
  is dropped with a warning, listed in the report, and scores and draws
  are recomputed without it.
- primary_metrics.csv gains a dropped column, so a report rebuilt by
  synthdata-plot keeps the anchored table and the left-out list.
- The dashboard's ranking footnote describes the anchored scores and the
  bootstrap ties instead of the Welch t-test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013mqpy2RQu95wcEnosoxLFj
@alperkent-cmi
alperkent-cmi marked this pull request as ready for review November 2, 2026 19:54
@alperkent-cmi
alperkent-cmi merged commit 37af728 into main Nov 2, 2026
6 checks passed
@alperkent-cmi
alperkent-cmi deleted the fix/ranking-anchor-noise branch November 2, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants