Repository navigation
Run the benchmark through compile_selected and stop describing distillation as the method - #78
Merged
Merged
Conversation
compile_selected implements the protocol of #75. Rows are split into Fit, Select and Report: stratified, out-of-time by date, or by group so one applicant lands in one partition. The ceiling (the teacher) is tuned once by the user's factory, frozen, and refit K times for cross-fitted soft targets; one search plus K fits, never K searches. Every (alpha, trees, depth, monotone) configuration is compiled on Fit and scored on Select as an integer artifact, so quantization and banding are priced too. The simplest configuration statistically tied with the best wins, preferring alpha = 1 as a governance prior recorded as the tie rule; the floor gates; the winner is refit on Fit union Select; result.report() reads Report once and writes the figures into a hash-covered provenance block, so what ships carries its own reported cost. Below a configurable event count in Select, nested cross-validation takes over with a warning. Whether the whitebox learns from the ceiling's probabilities or from the labels is now a selected parameter. Every step is also a plain function. compileml.batch.score_batch is the runtime's score path over a matrix in NumPy, asserted bit-identical to decide() on every row across calibration modes, input precision and missing values; 200,000 rows in under a second. Selection ranks compiled artifacts with it and never computes attribution. The trees axis is free: extract_trees passes an ExtractedModel through, so one fit at the largest tree count is extracted once and compiled from prefixes, verified equal to a separate fit for both backends. train_whitebox(backend="hist") selects the histogram backend explicitly, measured at 1-2 s where the classic backend took 18 s at 30k rows and 293 s at 300k; compile_selected uses it throughout, and the default is unchanged. Validation check 11, selection hygiene, reads the provenance block and lists broken protocol: soft targets not cross-fitted, partitions that overlap, Report read more than once. Advisory unless required, following check 10. sweep_whitebox warns when alpha_grid is not passed and when one holdout is used both to choose and to report. Part of #75; the documentation and the benchmark rerun follow separately.
…lation as the method The benchmark selects the whitebox on Select from the default grid of 40 configurations, refits the winner on Fit union Select, and reads retention once from Report. It chose alpha = 0.75, 80 trees, depth 2: a blend of labels and the ceiling's cross-fitted predictions, with both pure targets scoring lower on Select. Retention is 98.04% on Report with a 95% interval of 97.38 to 98.72, against 97.9% before on a holdout that had also chosen the configuration, and the artifact stands at 103.41% of a WoE logistic floor. The ceiling is the histogram GBM, fixed and recorded as unsearched, so the whole run takes minutes; the determinism check reruns the entire protocol and requires the same configuration and the same hash. results.json gains the selected configuration, intervals, floor figures, KS, Brier and events per split, and drops distill_spearman_train; the selection curve is committed beside it. The documentation stops describing distillation as what CompileML does. "Compile" is the verb; the strongest model is the ceiling (the teacher), a measuring instrument that never ships; retention is defined at first use as artifact Gini over ceiling Gini on Report, against a ceiling tuned under a declared budget; the floor is reported at equal weight. The README quick example runs through compile_selected, the quickstart keeps the by-hand path, the tuning guide gains "Let the data choose", the FAQ answers "the pipeline selects it", the validation pages count eleven checks, and the API reference lists the selection and batch-scoring functions. No API name changes. Closes #75
Every millisecond figure in the rerun is 1.2-1.5x slower than the 0.5.1 run on identical models with identical tree-walk counts: the reference laptop's state on the day, not the code. The changelog says so; the transcribed copies follow the committed file as they must.
benchmarks/sync_docs.py copies the committed figures into the README table, the attribution page's cost table and the two sentences that quote a latency. Each target must match exactly once or the script stops, and it is idempotent. RELEASING.md gains a section on refreshing the benchmark on an idle machine.
The previous commit landed the script before ruff had passed. Builtin open with context managers, one wrapped line, and the two mentions: the README says its table is written by the script, never by hand, and RELEASING.md gains a section on refreshing the benchmark on an idle machine.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second half of #75, stacked on #77. Merge #77 first; until then this diff also shows #77's commit.
The benchmark, through
compile_selectedThe whitebox is now chosen on Select from the default grid of 40 configurations, refit on Fit ∪ Select, and reported once on Report. The result is the most interesting number this benchmark has produced:
An interior blend won. On Select, the best pure-distillation configuration scored 0.6546, the best labels-only configuration 0.6568, and α = 0.75 with 160 trees 0.6679. Four configurations tied within one SE; the tie rule chose α = 0.75 with 80 trees over 160. At 16 events per split on the largest models, this is exactly the regime the RFC said soft targets should help in, and neither of the two old assumptions (distil, or don't) would have found it.
A second full run selected the same configuration and produced the same artifact hash; the determinism check now covers the whole protocol, not just the build.
Ceiling change. The benchmark's ceiling is now
HistGradientBoostingClassifier(300 trees, depth 4), recorded in provenance as a fixed, unsearched configuration. With the classic classifier the K refits would have taken ten minutes; the whole run now takes minutes, which #75 asked for.results.jsongains the selected configuration,retention_ci, floor figures, KS, Brier,eps_minand partition sizes, and dropsdistill_spearman_train, which had no meaning once the target is selected.benchmarks/selection_curve.jsonis new: every configuration scored on Select, labelled as ranking configurations rather than describing the shipped artifact.Latency: a caution, not a regression. Every millisecond figure in this run is 1.2–1.5× slower than the 0.5.1 run on models that are identical (same tree-walk counts). A probe on the same model swung between 0.86 and 2.04 ms within seconds, so the reference laptop was erratic today. The committed file is one run, and every transcribed copy (README table, attribution page, FAQ) follows it, as the 0.5.1 rule requires; the changelog says why the milliseconds moved. Worth a rerun on an idle machine before the release is tagged:
python benchmarks/run_benchmarks.py, thenpythonthe README table fromresults.json(the script is in the PR conversation), or ask me.Language
The documentation stops describing distillation as what CompileML does, per the RFC's migration table, prose only:
compile_selected; the benchmark section explains the protocol and the selected configuration; check 11 joins the validation list; the roadmap gains 0.9 and points at Gate experiment: cross-fitted α sweep with event thinning on public data — decides 0.10.0 #76.compile_selected, covering roles, partitions, cross-fitting, scoring as an artifact, one-SE-simplest, the floor gate, the final refit, nested CV, events per split as a diagnostic with the boundary called a lower bound, and cost. Choosing the target is rewritten around selection; the holdout paragraph explains the leak; Defaults leads withcompile_selected.--require-selection-hygiene.compile_selected,SelectionResult,make_partitions,default_grid,choose) andscore_batch.bands.md, the spec's aside, CONTRIBUTING's Translate compilation cost into expected-loss terms #19 mention. Translate compilation cost into expected-loss terms #19 itself is retitled to compilation cost.No API names change. Notebooks are untouched; the by-hand path they show is still valid.
Checks
ruff,black, fullpytestandmkdocs build --strictpass. Retention figures are deterministic and were reproduced by the benchmark's own second run.