Run the benchmark through compile_selected and stop describing distillation as the method

CHANGELOG.md

@@ -7,6 +7,57 @@ inside each artifact (`schema_version`).
 
 ## [Unreleased]
 
+### Added
+- `compile_selected` ([#75](https://github.com/orgoca/CompileML/issues/75)):
+  the target and the configuration are chosen on data the report never sees.
+  Rows are split into Fit, Select and Report; the ceiling (the teacher) is
+  tuned once, frozen and refit K times for cross-fitted soft targets; every
+  (α, trees, depth, monotone) configuration is compiled on Fit and scored on
+  Select as an integer artifact; the simplest configuration statistically
+  tied with the best wins, preferring α = 1 as a governance prior; the floor
+  gates; the winner is refit on Fit ∪ Select; and `result.report()` reads
+  Report once, writing the figures into a hash-covered provenance block. Below
+  a configurable event count in Select, nested cross-validation takes over.
+  Whether the whitebox learns from the ceiling's probabilities or from the
+  labels is now a selected parameter, not an assumption.
+- `compileml.batch.score_batch`: the runtime's score path over a matrix, in
+  NumPy, asserted bit-identical to `decide()` on every row — 200,000 rows in
+  under a second. Selection ranks compiled artifacts with it; deployments
+  still use the runtime or an export.
+- Validation check 11, selection hygiene: reads the provenance block and
+  lists broken protocol — soft targets not cross-fitted, partitions that
+  overlap, Report read more than once. Advisory unless
+  `require_selection_hygiene=True`, following check 10.
+- `train_whitebox(backend="hist")` selects the histogram backend explicitly.
+  It was measured at 1–2 s where the classic backend took 18 s at 30k rows and
+  293 s at 300k; `compile_selected` uses it throughout. The default is
+  unchanged, since changing it would change every artifact built with the
+  defaults.
+- `extract_trees` passes an `ExtractedModel` through, so a boosted fit can be
+  extracted once and compiled from prefixes of its trees: the first k trees of
+  a fit are the k-tree model, verified for both backends.
+
+### Changed
+- `sweep_whitebox` warns when `alpha_grid` is not passed — its default of
+  pure distillation is the assumption real data falsified — and when `X_val`
+  is used both to choose and to report. The defaults switch in 1.0.
+- The documentation stops describing distillation as what CompileML does.
+  "Compile" is the verb; the strongest model is the *ceiling* (the teacher),
+  a measuring instrument that never ships; *retention* is defined at first
+  use as artifact Gini over ceiling Gini on the Report partition, against a
+  ceiling tuned under a declared budget; the floor is reported at equal
+  weight. The README quick example runs through `compile_selected`, the
+  quickstart keeps the by-hand path, the tuning guide gains *Let the data
+  choose*, and the FAQ entry on distilling versus labels answers "the
+  pipeline selects it". API names are unchanged.
+- The benchmark runs through `compile_selected`. It selected α = 0.75, 80 trees, depth 2 on Select — both pure targets scored lower — and reports 98.04% retention on Report with a 95% interval of 97.38–98.72%, against 97.9% before on a holdout that had also chosen the configuration; the artifact stands at 103.41% of a WoE logistic floor (interval 102.54–104.25%). The ceiling is now the histogram GBM (300 trees, depth 4, fixed and recorded as unsearched), so the whole run takes minutes. `results.json` gains the selected configuration, intervals, floor figures, KS, Brier and events per split, and drops `distill_spearman_train`; the selection curve is committed as `benchmarks/selection_curve.json`. The determinism check now reruns the entire protocol and requires the same configuration and the same hash.
+
+  The latency block was re-measured in the same run and every millisecond figure moved by
+  1.2–1.5× on models identical to 0.5.1's, with identical tree-walk counts, so the change is
+  the reference laptop's state on the day, not the code. The transcribed copies in the README,
+  the attribution page and the FAQ follow the committed file, as they must; the walk counts
+  are the machine-independent measure and are unchanged.
+
 ## [0.8.0] - 2026-09-13
 
 What compilation cost where decisions are made — and what to do about it.

CONTRIBUTING.md

@@ -23,7 +23,7 @@ infrastructure and something a risk function could actually adopt:
 Smaller entry points: [#18](https://github.com/orgoca/CompileML/issues/18)
 (WOE compatibility docs), [#31](https://github.com/orgoca/CompileML/issues/31)
 (waterfall labels clipped at the canvas edge),
-[#19](https://github.com/orgoca/CompileML/issues/19) (distillation cost in
+[#19](https://github.com/orgoca/CompileML/issues/19) (compilation cost in
 expected-loss terms).
 
 Domain knowledge is as welcome as code. Several issues — segmented artifact

README.md

@@ -20,55 +20,42 @@ The choice is not unique to credit. It appears wherever a model makes a decision
 
 CompileML removes that choice.
 
-Train the strongest teacher you can: XGBoost, LightGBM, NN. CompileML projects its predictive structure into a shallow, integer-valued whitebox and packages the complete decision into one hashed artifact: score, calibrated probability, risk bands, reason codes, attribution.
+Train the strongest model you can: XGBoost, LightGBM, a neural network. It sets the ceiling. CompileML then searches for the best whitebox it can compile — trained on the outcomes, on the ceiling's predictions, or on a blend — and chooses on data it will not report on. The whole decision is packaged into one hashed artifact: score, calibrated probability, risk bands, reason codes, attribution. What you get is the artifact, what its guarantees cost against that ceiling, and whether it beats the scorecard you already have.
 
-Then the teacher is discarded. It was a means, not a deliverable, and it never reaches production.
+The ceiling is a measuring instrument. It never ships.
 
 What ships instead is decision logic built from integer addition, comparison, and table lookup. It runs under CompileML's standard-library-only Python implementation, an ordinary application, a small Lambda, a batch job or you export it as standalone SQL or COBOL and run it directly where the decision already happens. No XGBoost in production, no scikit-learn, no scoring service to operate. In exported form there is no model runtime at all.
 
-This is not a lighter way to serve the black box. The black box was used as a teacher and then compiled out of the system.
+This is not a lighter way to serve the black box. The black box set the bar and was then compiled out of the system.
 
 That buys three things, and I would not trade any one of them for the other two: the artifact reproduces to the integer on any machine, it explains itself by arithmetic rather than approximation, and it lands where the decision already runs. The sections below are how each one is enforced.
 
-The benchmark puts the cost at about 2% of the teacher's Gini. In exchange, at whitebox depth two or less, every decision reconstructs exactly from a printed scorecard table, and every explanation adds back to the production score with nothing left over.
+The benchmark puts the cost at about 2% of the ceiling's Gini — 98.04% retained on rows the selection never saw, with a 95% interval of 97.38–98.72% — and the artifact out-scores a WoE logistic regression on the same rows by 3.4%. In exchange, at whitebox depth two or less, every decision reconstructs exactly from a printed scorecard table, and every explanation adds back to the production score with nothing left over.
 
 The examples in this repository come from credit, where the project began and where lenders must explain every adverse decision, reproduce it for a validator, and often run it on systems that predate Python. [Where it fits](docs/concepts/where-it-fits.md) maps the vocabulary to other domains and says where the fit is weaker.
 
 ## Quick example
 
-Train however you want. The example below uses a strong model as a teacher and distills it into a shallow whitebox:
+Train the strongest model you can. CompileML chooses the whitebox, prices it against that model, and reads the reported figure from rows it never chose on:
 
 ```python
-from compileml.compile import train_whitebox
-from compileml.bands import monotone_quantile_bands
-from compileml.artifact import build_artifact, save_artifact
+from compileml import compile_selected
+from compileml.artifact import save_artifact
 
-whitebox, fidelity = train_whitebox(
-    X_train,
-    teacher.predict_proba(X_train)[:, 1],
-)
-
-latent = whitebox.predict(X_train).clip(0, 1)
-
-bands = monotone_quantile_bands(
-    latent,
-    y_train,
-    n_bands=10,
-)
-
-artifact = build_artifact(
-    whitebox,
-    feature_names,
-    baseline=medians,
-    band_edges=bands,
-    calibration_latent=latent,
-    calibration_y=y_train,
+result = compile_selected(
+    X, y,
+    ceiling=lambda X_fit, y_fit: XGBClassifier(**tuned).fit(X_fit, y_fit),
+    reference="woe",              # the floor: a WoE logistic regression, fitted inside Fit
     reasons=REASON_DICTIONARY,
 )
 
-save_artifact(artifact, "decision.json")
+result.selected                   # the target and configuration chosen on Select
+report = result.report()          # Report, read once: retention and floor ratio, with intervals
+save_artifact(result.artifact, "decision.json")
 ```
 
+Every step is also a plain function — `train_whitebox`, `monotone_quantile_bands`, `build_artifact` — for when you want each one in your hands; the [quickstart](docs/quickstart.md) walks that path.
+
 Production does not need the training stack:
 
 ```python
@@ -171,7 +158,7 @@ The artifact includes a SHA-256 hash. Loaders verify it by default and reject a
 
 ## Measured performance
 
-The committed benchmark uses deterministic synthetic credit data with 40,000 rows and 23 features. It runs on a consumer laptop through the pure-Python runtime.
+The committed benchmark uses deterministic synthetic credit data with 40,000 rows and 23 features, split 60/20/20 into Fit, Select and Report. `compile_selected` chooses the target, the tree count and the depth on Select from 40 configurations; the winner is refit on Fit ∪ Select; and every retention figure below is read once from Report. It selected α = 0.75 — a blend of labels and the ceiling's cross-fitted predictions — with 80 trees at depth 2, inside a tie band of 4; both pure targets, labels alone and the ceiling alone, scored lower on Select. The ceiling is a 300-tree, depth-4 GBM with a fixed, undeclared-search configuration, recorded as such. Latency runs on a consumer laptop through the pure-Python runtime.
 
 You can reproduce every number from an editable install of the checked-out source:
 
@@ -180,23 +167,25 @@ pip install -e .
 python benchmarks/run_benchmarks.py
 ```
 
-`results.json` records the CompileML version it measured, so a run against an older installed copy cannot pass for a measurement of the current code.
-
-| Metric                                     |                      Value |
-| ------------------------------------------ | -------------------------: |
-| Teacher Gini, 300-tree GBM                 |                      0.667 |
-| **Compiled integer artifact Gini**         | **0.653 — 97.9% retained** |
-| Band-ordinal Gini, 10 bands                |     0.647 — 97.0% retained |
-| Spearman correlation, teacher vs. artifact |                      0.977 |
-| Score + band + calibrated PD               |         **0.03 ms median** |
-| Score + band + calibrated PD, p95          |                    0.04 ms |
-| Full explained decision, 120 trees         |             0.62 ms median |
-| Full explained decision, p95               |                    0.84 ms |
-| Band assignment alone                      |                     0.2 µs |
-| Artifact size                              |                      97 KB |
-| Identical hash on rebuild                  |                        Yes |
-
-That 2% of Gini is the price of everything above it. It is stated rather than hidden, and it is reproducible on your own data with `compileml.tune.sweep_whitebox`.
+`results.json` records the CompileML version it measured, so a run against an older installed copy cannot pass for a measurement of the current code. The table below and the cost figures elsewhere in the docs are written from that file by `python benchmarks/sync_docs.py`, never by hand.
+
+| Metric                                         |                                  Value |
+| ---------------------------------------------- | -------------------------------------: |
+| Ceiling Gini, 300-tree GBM                     |                                  0.664 |
+| **Compiled integer artifact Gini**             | **0.651 — 98.0% retained (97.4–98.7)** |
+| Floor Gini, WoE logistic regression            |             0.630 — artifact at 103.4% |
+| Band-ordinal Gini, 10 bands                    |                 0.643 — 96.7% retained |
+| Spearman correlation, ceiling vs. artifact     |                                  0.980 |
+| Selected on Select, of 40 configurations       |            α = 0.75, 80 trees, depth 2 |
+| Score + band + calibrated PD                   |                     **0.03 ms median** |
+| Score + band + calibrated PD, p95              |                                0.06 ms |
+| Full explained decision, 80 trees              |                         0.70 ms median |
+| Full explained decision, p95                   |                                1.20 ms |
+| Band assignment alone                          |                                 0.4 µs |
+| Artifact size                                  |                                  70 KB |
+| Same configuration and identical hash on rerun |                                    Yes |
+
+That 2% of Gini is the price of everything above it. It is stated rather than hidden, it comes with an interval, and `compile_selected` measures it the same way on your own data. The full selection curve — every configuration scored on Select — is committed beside the results as `benchmarks/selection_curve.json`; it ranks configurations and does not describe the shipped artifact.
 
 One honest qualification: scoring is very fast; full explanation costs more.
 
@@ -206,10 +195,10 @@ On the benchmark's 120-tree ensemble, attribution alone:
 
 | features | perturbation | per tree | tree walks, perturbation | tree walks, per tree |
 | -------: | -----------: | -------: | -----------------------: | -------------------: |
-| 8 | 0.94 ms | 0.62 ms | 4,560 | 792 |
-| 23 | 6.79 ms | 0.70 ms | 33,360 | 912 |
-| 50 | 32.26 ms | 0.78 ms | 153,240 | 944 |
-| 100 | 131.31 ms | 0.76 ms | 606,240 | 936 |
+| 8 | 1.23 ms | 0.76 ms | 4,560 | 792 |
+| 23 | 9.07 ms | 0.89 ms | 33,360 | 912 |
+| 50 | 47.97 ms | 0.94 ms | 153,240 | 944 |
+| 100 | 200.69 ms | 0.95 ms | 606,240 | 936 |
 
 The walk counts are exact and hold on any machine; the milliseconds belong to one laptop. Cost follows walks, and walks follow tree structure rather than width: the sweep's models are fitted to a target that uses every feature, so more of their trees split on three distinct features than the headline model's do, which is why attribution alone at 23 features here costs slightly more than the full decision in the table above. At eight features the difference between the two derivations is modest. At a hundred it is the difference between a quadratic cost and a flat one. It is exact rather than sampled, and both derivations reach identical integers on every timed row.
 
@@ -240,7 +229,7 @@ For banding, `band_efficiency()` reports what the ladder discards — the Gini g
 
 ## What CompileML is not
 
-CompileML is not a new training framework. Use XGBoost, LightGBM, scikit-learn, or another teacher that can be distilled into the supported whitebox representation.
+CompileML is not a new training framework. Use XGBoost, LightGBM, scikit-learn, or any model that can set the ceiling for a whitebox, or be compiled into the supported representation directly.
 
 It is not a promise that your data pipelines are identical. Determinism means:
 
@@ -325,7 +314,8 @@ It checks:
 7. explanation stability;
 8. reason-code coverage;
 9. declared monotone directions, re-verified against the shipped trees;
-10. that the artifact out-scores a reference model on the same data.
+10. that the artifact out-scores a reference model on the same data;
+11. that, when the artifact records how it was selected, the protocol was followed: cross-fitted soft targets, disjoint partitions, one Report read.
 
 These checks run against the compiled artifact through the same runtime used for production decisions. There is no separate notebook implementation allowed to become "almost the same" over time.
 
@@ -413,12 +403,14 @@ Recently shipped:
 * exact attribution aggregated per tree, so explaining a decision no longer grows with feature count, and a fairness audit in `compileml.fairness` (0.5);
 * exact drift decomposition, band calibration and baseline staleness in `compileml.monitor` (0.6);
 * the calibrated PD and reason codes from the COBOL and SQL exports (0.7);
-* retention by segment and across PD cutoff ranges, and weighting a whitebox toward the segment that pays (0.8).
+* retention by segment and across PD cutoff ranges, and weighting a whitebox toward the segment that pays (0.8);
+* `compile_selected`: the target and the configuration chosen on data the report never sees, the ceiling as yardstick, a provenance block that carries the reported cost, and a NumPy batch scorer (0.9).
 
 The current priorities are:
 
-* add Java and C exporters;
-* add an optional NumPy batch scorer.
+* the gate experiment ([#76](https://github.com/orgoca/CompileML/issues/76)): where soft targets stop helping, and whether log-loss earns a logit-space latent;
+* a logit-space latent, if that experiment supports it;
+* add Java and C exporters.
 
 ## License
 

RELEASING.md

@@ -22,6 +22,21 @@ tokens stored anywhere.
    https://pypi.org/simple compileml` in a scratch venv, then
    `compileml --help` and a quick `import compileml`.
 
+## Refreshing the benchmark
+
+Run it on an idle machine — the millisecond figures belong to the laptop's state
+on the day — then let the script copy the numbers into the docs:
+
+```bash
+python benchmarks/run_benchmarks.py
+python benchmarks/sync_docs.py
+```
+
+Commit `benchmarks/results.json`, `benchmarks/selection_curve.json` and the
+documents the script touched together. Retention figures are deterministic
+and should not move between runs; if they do, something other than the
+machine changed.
+
 ## Releasing vX.Y.Z
 
 1. Bump `__version__` in `src/compileml/__init__.py` — the single source of