Repository navigation
fix(ranking): leave out primary metrics whose baselines are within noise; dashboard footnote (v0.31.1) - #150
Merged
Merged
Conversation
…ise; dashboard footnote; v0.31.1 - Anchoring divides by the baselines' gap; when that gap is within its own noise the scores blow up (fixture: fairness interval down to -47). With the ranking bootstrap on, a metric is anchored only when the one-sided 95% lower bound of the baseline gap is above 0; otherwise it is dropped with a warning, listed in the report, and scores and draws are recomputed without it. - primary_metrics.csv gains a dropped column, so a report rebuilt by synthdata-plot keeps the anchored table and the left-out list. - The dashboard's ranking footnote describes the anchored scores and the bootstrap ties instead of the Welch t-test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013mqpy2RQu95wcEnosoxLFj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Alp · project thread
What (v0.31.1)
Two follow-ups to the anchored ranking (v0.30.0), stacked on the model-cards PR (v0.31.0).
demographic_parity_gap_error, whose gap was above 0 in only 13% of draws. The reasoning is indocs/verification.md(Baseline-anchored scores).synthdata-plotkeeps the anchored primary table and the left-out list;primary_metrics.csvgains adroppedcolumn.Checks
uv run ty check, ruff check/format, unit tests (896 passed), CPU integration tier (91 passed).🤖 Generated with Claude Code
https://claude.ai/code/session_013mqpy2RQu95wcEnosoxLFj