diff --git a/CHANGELOG.md b/CHANGELOG.md index 33bb64c..eabe1a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ inside each artifact (`schema_version`). ## [Unreleased] +## [0.9.0] - 2026-09-27 + +The teacher as yardstick, the target as a selected parameter. + +Distillation was the method. On real data it lost to the same whitebox +trained on labels, so it is no longer assumed anywhere. `compile_selected` +trains candidates on labels, on the ceiling's cross-fitted predictions and on +blends; selects the target, tree count and depth on rows it will not report +on; refits the winner on Fit ∪ Select; and reads Report once, writing the +figures — and how they were produced — into the artifact's hashed provenance. +The strongest model is the ceiling, a measuring instrument that prices +compilation and never ships. The floor is reported at equal weight. + +The benchmark, run this way, selected a blend — α = 0.75, 80 trees, depth 2 — +with both pure targets scoring lower on Select. It reports 98.04% retention on +Report with a 95% interval of 97.38–98.72, and the artifact at 103.41% of a +WoE logistic floor. + +A minor release: `compile_selected`, `compileml.batch.score_batch`, +`train_whitebox(backend=)` and validation check 11 are new public API. The +runtime, exporters and artifact schema are unchanged. Probability space and +squared error only; the logit latent is gated on +[#76](https://github.com/orgoca/CompileML/issues/76). + ### Added - `compile_selected` ([#75](https://github.com/orgoca/CompileML/issues/75)): the target and the configuration are chosen on data the report never sees. @@ -52,11 +76,13 @@ inside each artifact (`schema_version`). pipeline selects it". API names are unchanged. - The benchmark runs through `compile_selected`. It selected α = 0.75, 80 trees, depth 2 on Select — both pure targets scored lower — and reports 98.04% retention on Report with a 95% interval of 97.38–98.72%, against 97.9% before on a holdout that had also chosen the configuration; the artifact stands at 103.41% of a WoE logistic floor (interval 102.54–104.25%). The ceiling is now the histogram GBM (300 trees, depth 4, fixed and recorded as unsearched), so the whole run takes minutes. `results.json` gains the selected configuration, intervals, floor figures, KS, Brier and events per split, and drops `distill_spearman_train`; the selection curve is committed as `benchmarks/selection_curve.json`. The determinism check now reruns the entire protocol and requires the same configuration and the same hash. - The latency block was re-measured in the same run and every millisecond figure moved by - 1.2–1.5× on models identical to 0.5.1's, with identical tree-walk counts, so the change is - the reference laptop's state on the day, not the code. The transcribed copies in the README, - the attribution page and the FAQ follow the committed file, as they must; the walk counts - are the machine-independent measure and are unchanged. + Latency is now measured in a fresh process after a 30 s rest. Selection refits the ceiling + with a multithreaded booster and compiles forty candidates, and timings taken in the same + process straight afterwards were 1.3–2.3× slower than the same artifact measured on its own, + from lingering worker threads and a hot CPU. On the cost sweep's models, identical to 0.5.1's + with identical tree-walk counts, the per-tree timings now come out at 0.94–0.98× the + 0.5.1 figures. The transcribed copies in the README, the attribution page and the FAQ are + written from the committed file by `benchmarks/sync_docs.py`. ## [0.8.0] - 2026-09-13 diff --git a/CITATION.cff b/CITATION.cff index b8976fa..f3a45dc 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -19,8 +19,8 @@ authors: repository-code: "https://github.com/orgoca/CompileML" url: "https://github.com/orgoca/CompileML" license: Apache-2.0 -version: 0.8.0 -date-released: "2026-09-13" +version: 0.9.0 +date-released: "2026-09-27" keywords: - credit risk - credit scoring diff --git a/README.md b/README.md index ab951ac..a499941 100644 --- a/README.md +++ b/README.md @@ -177,11 +177,11 @@ python benchmarks/run_benchmarks.py | Band-ordinal Gini, 10 bands | 0.643 — 96.7% retained | | Spearman correlation, ceiling vs. artifact | 0.980 | | Selected on Select, of 40 configurations | α = 0.75, 80 trees, depth 2 | -| Score + band + calibrated PD | **0.03 ms median** | -| Score + band + calibrated PD, p95 | 0.06 ms | -| Full explained decision, 80 trees | 0.70 ms median | -| Full explained decision, p95 | 1.20 ms | -| Band assignment alone | 0.4 µs | +| Score + band + calibrated PD | **0.02 ms median** | +| Score + band + calibrated PD, p95 | 0.03 ms | +| Full explained decision, 80 trees | 0.42 ms median | +| Full explained decision, p95 | 0.68 ms | +| Band assignment alone | 0.2 µs | | Artifact size | 70 KB | | Same configuration and identical hash on rerun | Yes | @@ -195,10 +195,10 @@ On the benchmark's 120-tree ensemble, attribution alone: | features | perturbation | per tree | tree walks, perturbation | tree walks, per tree | | -------: | -----------: | -------: | -----------------------: | -------------------: | -| 8 | 1.23 ms | 0.76 ms | 4,560 | 792 | -| 23 | 9.07 ms | 0.89 ms | 33,360 | 912 | -| 50 | 47.97 ms | 0.94 ms | 153,240 | 944 | -| 100 | 200.69 ms | 0.95 ms | 606,240 | 936 | +| 8 | 0.94 ms | 0.60 ms | 4,560 | 792 | +| 23 | 6.87 ms | 0.69 ms | 33,360 | 912 | +| 50 | 34.01 ms | 0.74 ms | 153,240 | 944 | +| 100 | 133.88 ms | 0.74 ms | 606,240 | 936 | The walk counts are exact and hold on any machine; the milliseconds belong to one laptop. Cost follows walks, and walks follow tree structure rather than width: the sweep's models are fitted to a target that uses every feature, so more of their trees split on three distinct features than the headline model's do, which is why attribution alone at 23 features here costs slightly more than the full decision in the table above. At eight features the difference between the two derivations is modest. At a hundred it is the difference between a quadratic cost and a flat one. It is exact rather than sampled, and both derivations reach identical integers on every timed row. @@ -221,7 +221,8 @@ Both are measurable rather than guessable: ```python from compileml.tune import sweep_whitebox, sweep_bands -sweep_whitebox(X, teacher_latent, y, X_val=X_val, y_val=y_val, teacher_latent_val=t_val) +sweep_whitebox(X, teacher_latent, y, alpha_grid=(0.0, 0.5, 1.0), + X_val=X_val, y_val=y_val, teacher_latent_val=t_val) sweep_bands(latent, y, k_grid=(4, 6, 8, 10, 12, 16)) ``` diff --git a/benchmarks/results.json b/benchmarks/results.json index 63214fd..5a46428 100644 --- a/benchmarks/results.json +++ b/benchmarks/results.json @@ -1,6 +1,6 @@ { "seed": 42, - "compileml": "0.8.0", + "compileml": "0.9.0", "python": "3.14.3", "machine": "Intel64 Family 6 Model 170 Stepping 4, GenuineIntel", "os": "Windows 11", @@ -48,36 +48,36 @@ "n_test": 8000 }, "latency": { - "score_band_pd_median_ms": 0.034, - "score_band_pd_p95_ms": 0.062, - "full_explain_median_ms": 0.696, - "full_explain_p95_ms": 1.197, - "band_ladder_median_us": 0.4, + "score_band_pd_median_ms": 0.021, + "score_band_pd_p95_ms": 0.025, + "full_explain_median_ms": 0.416, + "full_explain_p95_ms": 0.679, + "band_ladder_median_us": 0.2, "explain_by_features": { "8": { - "per_tree_ms": 0.756, - "perturbation_ms": 1.232, + "per_tree_ms": 0.602, + "perturbation_ms": 0.935, "per_tree_walks": 792, "perturbation_walks": 4560, "features_split_on": 8 }, "23": { - "per_tree_ms": 0.889, - "perturbation_ms": 9.068, + "per_tree_ms": 0.692, + "perturbation_ms": 6.87, "per_tree_walks": 912, "perturbation_walks": 33360, "features_split_on": 23 }, "50": { - "per_tree_ms": 0.94, - "perturbation_ms": 47.965, + "per_tree_ms": 0.736, + "perturbation_ms": 34.007, "per_tree_walks": 944, "perturbation_walks": 153240, "features_split_on": 50 }, "100": { - "per_tree_ms": 0.954, - "perturbation_ms": 200.695, + "per_tree_ms": 0.744, + "perturbation_ms": 133.883, "per_tree_walks": 936, "perturbation_walks": 606240, "features_split_on": 88 @@ -90,12 +90,13 @@ "measures": "attribution only, per row: median ms, and tree walks", "derivations_agree": true }, + "measured_in": "a fresh process, after a 30 s rest following selection", "note": "pure-Python stdlib runtime, single row, 80 depth-2 trees at p=23. full_explain is the whole decide() payload: score, band, PD, exact attribution and reason codes. Attribution is aggregated per tree, so its cost is O(trees) and independent of feature count; explain_by_features times the attribution alone by both derivations, with tree walks per row as the machine-independent measure." }, "artifact": { "size_kb": 70.2, "n_trees": 80, - "hash": "dcabaa0dd2e99ad0\u2026", + "hash": "465c984e8ed5ebdd\u2026", "exact_attribution": true }, "determinism": { diff --git a/benchmarks/run_benchmarks.py b/benchmarks/run_benchmarks.py index d20aaba..cf086ca 100644 --- a/benchmarks/run_benchmarks.py +++ b/benchmarks/run_benchmarks.py @@ -17,6 +17,8 @@ import json import platform import statistics +import subprocess +import sys import time from pathlib import Path @@ -120,7 +122,7 @@ def explain_cost_by_features() -> tuple[dict, dict]: "perturbation_walks": (2 + p + p * (p - 1) // 2) * len(split_sets), "features_split_on": len(set().union(*split_sets)), } - print(f" p={p:>3}: {by_p[str(p)]}") + print(f" p={p:>3}: {by_p[str(p)]}", file=sys.stderr) config = { "n_trees": SWEEP_TREES, @@ -139,6 +141,69 @@ def median_p95(samples_ms: list[float]) -> tuple[float, float]: ) +def measure_latency(artifact_path: Path, rows_path: Path) -> dict: + """Single-row latency and the explanation-cost sweep, in a fresh interpreter. + + Run as a child process after the selection phase. Selection refits the + ceiling with a multithreaded booster and compiles forty candidates; measured + in the same process straight afterwards, every single-threaded timing came + out 1.3-2.3x slower than the same artifact measured on its own, from + lingering worker threads and a hot CPU. A new process after a short rest + has neither. + """ + artifact = load_artifact(artifact_path) + rows_test = [[float(v) for v in row] for row in np.load(rows_path)] + decisions = [decide(artifact, row, explain=False) for row in rows_test] + + reps = 400 + score_ms = [] + for i in range(reps): + row = rows_test[i % len(rows_test)] + t0 = time.perf_counter() + decide(artifact, row, explain=False) + score_ms.append((time.perf_counter() - t0) * 1000) + + explain_ms = [] + for i in range(60): + row = rows_test[i % len(rows_test)] + t0 = time.perf_counter() + decide(artifact, row, explain=True) + explain_ms.append((time.perf_counter() - t0) * 1000) + + edges = artifact["bands"]["edges_int"] + band_us = [] + for i in range(reps): + latent_int = decisions[i % len(decisions)]["latent_int"] + t0 = time.perf_counter() + band_index(latent_int, edges) + band_us.append((time.perf_counter() - t0) * 1_000_000) + + by_features, by_features_config = explain_cost_by_features() + + n_trees = len(artifact["model"]["trees"]) + p = len(artifact["features"]["names"]) + s_med, s_p95 = median_p95(score_ms) + e_med, e_p95 = median_p95(explain_ms) + return { + "score_band_pd_median_ms": round(s_med, 3), + "score_band_pd_p95_ms": round(s_p95, 3), + "full_explain_median_ms": round(e_med, 3), + "full_explain_p95_ms": round(e_p95, 3), + "band_ladder_median_us": round(statistics.median(band_us), 2), + "explain_by_features": by_features, + "explain_by_features_config": by_features_config, + "measured_in": "a fresh process, after a 30 s rest following selection", + "note": ( + f"pure-Python stdlib runtime, single row, {n_trees} depth-2 trees at " + f"p={p}. full_explain is the whole decide() payload: score, band, PD, " + "exact attribution and reason codes. Attribution is aggregated per " + "tree, so its cost is O(trees) and independent of feature count; " + "explain_by_features times the attribution alone by both derivations, " + "with tree walks per row as the machine-independent measure." + ), + } + + def main() -> None: rng = np.random.default_rng(SEED) results: dict = { @@ -250,53 +315,22 @@ def gini(score) -> float: ) # ------------------------------------------------------------ latency - print("measuring latency…") - reps = 400 - score_ms = [] - for i in range(reps): - row = rows_test[i % len(rows_test)] - t0 = time.perf_counter() - decide(artifact, row, explain=False) - score_ms.append((time.perf_counter() - t0) * 1000) - - explain_ms = [] - for i in range(60): - row = rows_test[i % len(rows_test)] - t0 = time.perf_counter() - decide(artifact, row, explain=True) - explain_ms.append((time.perf_counter() - t0) * 1000) - - edges = artifact["bands"]["edges_int"] - band_us = [] - for i in range(reps): - latent_int = decisions[i % len(decisions)]["latent_int"] - t0 = time.perf_counter() - band_index(latent_int, edges) - band_us.append((time.perf_counter() - t0) * 1_000_000) - - print("measuring explanation cost by feature count…") - by_features, by_features_config = explain_cost_by_features() - - n_trees = len(artifact["model"]["trees"]) - s_med, s_p95 = median_p95(score_ms) - e_med, e_p95 = median_p95(explain_ms) - results["latency"] = { - "score_band_pd_median_ms": round(s_med, 3), - "score_band_pd_p95_ms": round(s_p95, 3), - "full_explain_median_ms": round(e_med, 3), - "full_explain_p95_ms": round(e_p95, 3), - "band_ladder_median_us": round(statistics.median(band_us), 2), - "explain_by_features": by_features, - "explain_by_features_config": by_features_config, - "note": ( - f"pure-Python stdlib runtime, single row, {n_trees} depth-2 trees at " - f"p={p}. full_explain is the whole decide() payload: score, band, PD, " - "exact attribution and reason codes. Attribution is aggregated per " - "tree, so its cost is O(trees) and independent of feature count; " - "explain_by_features times the attribution alone by both derivations, " - "with tree walks per row as the machine-independent measure." - ), - } + rows_path = HERE / "benchmark_rows.npy" + np.save(rows_path, X[parts.report]) + print("resting 30 s, then measuring latency in a fresh process…") + time.sleep(30) + child = subprocess.run( + [sys.executable, str(Path(__file__).resolve()), "--measure", str(path), str(rows_path)], + capture_output=True, + text=True, + encoding="utf-8", + check=True, + ) + results["latency"] = json.loads(child.stdout.strip().splitlines()[-1]) + rows_path.unlink() + for line in child.stderr.splitlines(): + if line.startswith(" p="): + print(line) # ------------------------------------------------------------ artifact results["artifact"] = { @@ -385,4 +419,7 @@ def gini(score) -> float: if __name__ == "__main__": - main() + if len(sys.argv) > 1 and sys.argv[1] == "--measure": + print(json.dumps(measure_latency(Path(sys.argv[2]), Path(sys.argv[3])))) + else: + main() diff --git a/docs/concepts/attribution.md b/docs/concepts/attribution.md index 552f8cd..7633ab6 100644 --- a/docs/concepts/attribution.md +++ b/docs/concepts/attribution.md @@ -60,10 +60,10 @@ From the committed benchmark, attribution alone on a 120-tree ensemble: | features | perturbation | per tree | tree walks, perturbation | tree walks, per tree | |---|---|---|---|---| -| 8 | 1.23 ms | 0.76 ms | 4,560 | 792 | -| 23 | 9.07 ms | 0.89 ms | 33,360 | 912 | -| 50 | 47.97 ms | 0.94 ms | 153,240 | 944 | -| 100 | 200.69 ms | 0.95 ms | 606,240 | 936 | +| 8 | 0.94 ms | 0.60 ms | 4,560 | 792 | +| 23 | 6.87 ms | 0.69 ms | 33,360 | 912 | +| 50 | 34.01 ms | 0.74 ms | 153,240 | 944 | +| 100 | 133.88 ms | 0.74 ms | 606,240 | 936 | Flat, not merely faster. The walk counts are exact and hold on any machine; the milliseconds are one laptop's. The benchmark's target spreads signal over @@ -87,7 +87,7 @@ attributions rather than sample estimates with selection effects. This is why `decide()` defaults to `explain=True`. Batch re-explanation used to be the place the cost hurt: 10M accounts at -8.4 ms is roughly 23 CPU-hours, and it grew with p². At 0.70 ms it is under +8.4 ms is roughly 23 CPU-hours, and it grew with p². At 0.42 ms it is under two, and it no longer grows with feature count. That is what makes reason-code emission in the SQL export tractable rather than impractical. Live latency was never the constraint. diff --git a/docs/faq.md b/docs/faq.md index b8f6c86..9a27ffc 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -169,7 +169,7 @@ labels. It is still more work than scoring, but no longer dramatically so, and no longer quadratic in feature count. Attribution is aggregated per tree, which makes the cost `O(trees)` and independent of `p` — on the committed -benchmark's 120-tree ensemble, 0.8–1.0 ms per row whether the model has +benchmark's 120-tree ensemble, 0.6–0.7 ms per row whether the model has 8 features or 100. More trees cost proportionally more; more features do not. That is real-time for credit decisioning, which is why diff --git a/docs/howto/tuning.md b/docs/howto/tuning.md index 5f4cc80..24fea9e 100644 --- a/docs/howto/tuning.md +++ b/docs/howto/tuning.md @@ -115,6 +115,7 @@ rows = sweep_whitebox( X_train, teacher_latent_train, y_train, trees_grid=(20, 40, 80, 160, 320), depth_grid=(1, 2), + alpha_grid=(0.0, 0.5, 1.0), # the target is a choice; say which X_val=X_val, y_val=y_val, teacher_latent_val=teacher_latent_val, ) # pandas.DataFrame(rows) if you like tables @@ -470,8 +471,10 @@ two backends also regularize differently (histogram binning, leaf-size defaults), so do not guess the cost — measure it: ```python -rows_free = sweep_whitebox(X, latent, y, X_val=Xv, y_val=yv, teacher_latent_val=lv) -rows_mono = sweep_whitebox(X, latent, y, X_val=Xv, y_val=yv, teacher_latent_val=lv, +rows_free = sweep_whitebox(X, latent, y, alpha_grid=(0.0, 1.0), + X_val=Xv, y_val=yv, teacher_latent_val=lv) +rows_mono = sweep_whitebox(X, latent, y, alpha_grid=(0.0, 1.0), + X_val=Xv, y_val=yv, teacher_latent_val=lv, monotone_constraints={"UTIL": +1, "TENURE": -1}) ``` diff --git a/examples/README.md b/examples/README.md index 914c61b..6750a0a 100644 --- a/examples/README.md +++ b/examples/README.md @@ -10,7 +10,7 @@ by `tests/test_viz.py`. Re-run it after changing anything in `compileml.viz`. | Notebook | What it shows | |---|---| -| [`01_quickstart.ipynb`](01_quickstart.ipynb) | The full pipeline: teacher → whitebox → bands → reason codes → hashed artifact → stdlib-runtime decision → eight-check validation. Includes the reconciliation identity re-added by hand. | +| [`01_quickstart.ipynb`](01_quickstart.ipynb) | The full pipeline: teacher → whitebox → bands → reason codes → hashed artifact → stdlib-runtime decision → validation. Includes the reconciliation identity re-added by hand. | | [`02_deploy_sql_cobol.ipynb`](02_deploy_sql_cobol.ipynb) | The same artifact as a generated SQL query — **executed in SQLite and diffed against the Python runtime, integer for integer** — and as a generated COBOL program. | | [`03_governance.ipynb`](03_governance.ipynb) | Tamper refusal (edit one integer, loading fails), zero-churn recalibration with the predecessor-hash chain, and certified bands that find five real risk plateaus but refuse to segment noise. | | [`04_visualization.ipynb`](04_visualization.ipynb) | Reference gallery for `compileml[viz]`: waterfall arrow geometry, exact remainder truncation, the depth>2 residual bar, dependency-free SVG, every `color_by` / `value_color` / `sort_metric` option, per-band facets, and restyling with your own palette. | diff --git a/src/compileml/__init__.py b/src/compileml/__init__.py index 4673c6d..8de878d 100644 --- a/src/compileml/__init__.py +++ b/src/compileml/__init__.py @@ -16,7 +16,7 @@ # this attribute at build time (setuptools dynamic version), so the wheel # metadata, `compileml.__version__`, `compileml inspect`, and the # `compileml_version` recorded inside every artifact can never disagree. -__version__ = "0.8.0" +__version__ = "0.9.0" __all__ = ["compile_selected", "decide", "load_artifact", "verify_artifact", "__version__"]