[alperkent-cmi]

Requested by Alp · project thread

Summary

Adds notebooks/classification_metrics.py, an interactive marimo notebook for entry-level data scientists on the classifier scores the pipeline reports. Version 0.31.2, documentation only: no pipeline code, config or output changes.

• Uses a generic three-class readmission example with an 82% / 13% / 5% split (1,000 test patients).
• Explains threshold-dependent metrics (macro-F1, balanced accuracy, Cohen's kappa default and with inverse-frequency class weights, accuracy as a warning) against ranking metrics (macro AUROC, macro AUPRC), with formulas, chance levels, strengths and weaknesses. Shows that class-weighted kappa equals (balanced accuracy - 1/K) / (1 - 1/K).
• Nine confusion-matrix scenarios (always majority, the same with informative scores, two random guessers, high precision, high sensitivity with many false alarms, a balanced model, early-vs-late near misses, near perfect), with exact threshold metrics and a heatmap.
• Per-class ROC and precision-recall curves with chance lines, for any scenario, under the separation slider and in the playground. The metric heatmap uses the coolwarm colour map.
• A worked calculation, a threshold-shift slider (threshold metrics move, AUPRC does not), a score-separation slider (the reverse), and an editable confusion matrix that computes every metric.
• AUROC and AUPRC need scores, so the notebook simulates class probabilities whose argmax reproduces each matrix exactly (fixed seed) and says so.
• Metric definitions match synthdata/evaluation/tstr.py (macro AUPRC = mean one-vs-rest average precision over present classes).

marimo is already a runtime dependency, so nothing is added.

Run: uv run marimo edit notebooks/classification_metrics.py

Checks

uv run ty check (0 errors), uvx ruff check ., uvx ruff format --check ., unit tests (858 passed), CPU integration tier, marimo check on the notebook, and the notebook run in a browser with marimo run.

🤖 Generated with Claude Code

https://claude.ai/code/session_013mqpy2KSiyZJUxDUxTBELL