Discussion: segmented scorecards as one artifact — should CompileML specify a suite? #13

Status: open for discussion — not a spec, not scheduled. This issue sets out a proposal so there is something concrete to react to. Whether CompileML should ever specify it is the question being asked, and the answer should come from people who have governed segmented scorecards in production, not from the maintainers alone. Disagreement, including "this should stay out of scope", is as useful as agreement.

Problem

Real portfolios run segmented scorecards: thin file vs thick file, new vs existing customer, by product. One CompileML artifact governs one population. There is currently no answer to:

• How do I govern a suite of six artifacts as one policy?
• How do I route an applicant to the right artifact, deterministically, with the routing itself auditable?
• How do bands stay comparable across segments (does G07 mean the same risk in both)?
• What does a champion/challenger comparison look like across a suite?

The obvious answer — "just make N artifacts" — loses the property that makes CompileML worth using: a single hashed object that is the decision. If routing lives in someone's Python glue, the glue is now an unaudited part of the decision, which is precisely the failure mode this project exists to prevent.

Why this has become more pressing

retention_by_segment (#12) can now show that one segment paid for compression — lower retention, more decisions that differ from the teacher across its cutoff range. The levers inside a single artifact have limits, measured on synthetic data built for each case (a 10% segment, depth 2):

Weighting a shared model makes segments compete for one budget: helping one costs the others. And an interaction that exists only in one segment is a three-way effect (segment × A × B), which no number of depth-2 trees expresses. A separate model per segment is the standard answer in scorecard practice. The question is how to have it without losing the single hashed decision.

Options

1. Suite artifact — one hashed document with deterministic routing rules and one full model per segment. Sketched below.
2. Segment as a feature — one artifact, segment indicator as an input. Simple and already possible; expresses segment × feature effects at depth 2, but not segment-only interactions, and every retrain touches every segment.
3. Out of scope — routing is the caller's responsibility, and the documentation states the boundary honestly.

A sketch of option 1, to react to

Shape. One artifact containing:

• routing rules — ordered, first-match comparisons on input features, with a mandatory catch-all branch, inside the hash;
• branches — each a complete depth-≤2 model with its own baseline and calibration, and its own hash;
• one band ladder on the PD scale shared by every branch, so a band means the same risk whichever branch decided and one policy cutoff applies across the suite.

What it would preserve. Routing is integer comparison and each branch is an ordinary depth-2 model, so the inference side stays standard library and simple arithmetic. SQL and COBOL exports become a CASE / EVALUATE on the route followed by the branch. Attribution stays exact per branch; reason codes come from the branch that decided, relative to that branch's baseline — as they do from whichever scorecard applied in a segmented suite today.

What it would add. Per-branch hashes would make change control provable: retrain the thin-file branch and the thick-file branch's hash is unchanged, so those decisions can be certified untouched without re-validating them.

What it would cost. A new artifact schema version. It would be sensible to bundle it with other spec changes (#39, #11) rather than bump the schema repeatedly.

Open questions

These are the ones we cannot answer well ourselves.

1. Is this CompileML's job at all? Or does routing belong to the decision engine around the artifact, as option 3 says?
2. What did your model risk function require of a segmented suite? Was the suite approved as one model or as N? Would per-branch hashes actually satisfy change control, or would any change re-open the whole suite?
3. Route on inputs only, or also on scores? Routing on an input (file depth, months on book, product) is sound. Routing on a score — model 1 scores everyone, model 2 re-decides applicants inside a score range — is used as a cascade in practice, but at the boundary it can invert risk ordering (a slightly riskier applicant crosses over and receives a lower PD), breaks monotone constraints, and gives reasons from a model that did not decide who was routed to it. If you have run cascades, how were those handled?
4. Shared band ladder or per-branch ladders? A shared PD ladder keeps one policy cutoff meaningful; per-branch ladders fit each segment better. Which did governance accept?
5. Minimum branch size. Small branches overfit training and calibration. Should the spec enforce a minimum count of rows and defaults per branch, and what is a defensible one?
6. Fairness. Routing variables must be permitted inputs, and segmentation can still recreate disparate treatment through a proxy. Is a per-branch fairness audit enough, or does the routing rule itself need its own review?
7. Champion/challenger across a suite. Should a branch be replaceable on its own (Champion/challenger: diff two artifacts at the band and decision level #16), and what does the comparison report?

Ask

Discussion first, no PR. If you have governed segmented scorecards, the most valuable contribution here is what actually happened: what was required, what broke, and what you would not do again.