Repository navigation
Select the target and the configuration on data the report never sees (compile_selected) - #77
Merged
Merged
Conversation
compile_selected implements the protocol of #75. Rows are split into Fit, Select and Report: stratified, out-of-time by date, or by group so one applicant lands in one partition. The ceiling (the teacher) is tuned once by the user's factory, frozen, and refit K times for cross-fitted soft targets; one search plus K fits, never K searches. Every (alpha, trees, depth, monotone) configuration is compiled on Fit and scored on Select as an integer artifact, so quantization and banding are priced too. The simplest configuration statistically tied with the best wins, preferring alpha = 1 as a governance prior recorded as the tie rule; the floor gates; the winner is refit on Fit union Select; result.report() reads Report once and writes the figures into a hash-covered provenance block, so what ships carries its own reported cost. Below a configurable event count in Select, nested cross-validation takes over with a warning. Whether the whitebox learns from the ceiling's probabilities or from the labels is now a selected parameter. Every step is also a plain function. compileml.batch.score_batch is the runtime's score path over a matrix in NumPy, asserted bit-identical to decide() on every row across calibration modes, input precision and missing values; 200,000 rows in under a second. Selection ranks compiled artifacts with it and never computes attribution. The trees axis is free: extract_trees passes an ExtractedModel through, so one fit at the largest tree count is extracted once and compiled from prefixes, verified equal to a separate fit for both backends. train_whitebox(backend="hist") selects the histogram backend explicitly, measured at 1-2 s where the classic backend took 18 s at 30k rows and 293 s at 300k; compile_selected uses it throughout, and the default is unchanged. Validation check 11, selection hygiene, reads the provenance block and lists broken protocol: soft targets not cross-fitted, partitions that overlap, Report read more than once. Advisory unless required, following check 10. sweep_whitebox warns when alpha_grid is not passed and when one holdout is used both to choose and to report. Part of #75; the documentation and the benchmark rerun follow separately.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The code half of #75. The docs, README and benchmark rerun follow in a second PR, because they depend on the number this produces.
What it does
make_partitions: stratified 60/20/20;date=makes Report the latest slice;group=keeps one applicant in one partition; explicit index dicts are checked for overlap.row_id=overlap is a hard failure. Content-identical rows across partitions are counted and fail only aboveduplicate_threshold(0.01), since they are legitimate in credit data.best_estimator_, or the model itself), and refit K times for out-of-fold soft targets.ceiling_oof=/ceiling_scores=for users who only have predictions; provenance then saysuser_asserted.ceiling_budget=is recorded verbatim so retention is comparable across artifacts.tie_seSEs ties; among ties prefer α = 1, then fewer trees, then lower depth. Recorded as a governance prior.selection_metric="brier"is opt-in.allow_below_floor=True.metadata.provenance.reportand the artifact rehashed. A second call raises. This is hygiene, not proof, and the docstring says so.min_select_events(300), with a warning. In nested mode the factory is called once per outer fold and once for the final refit, since correctness matters more than cost on data that small.eps_min, and the report.Compute criteria from #75, each met
train_whitebox(backend="hist"), new and explicit; the default is unchangedextract_treespasses anExtractedModelthrough;test_prefix_of_a_fit_is_the_smaller_fittest_ceiling_is_tuned_once_and_refit_frozencounts: 1 factory call, 1 + K + 1 fits (+ K when α < 1 is chosen)decide()compileml.batch.score_batch; 0 mismatches across linear/step/no calibration, float32, and missing values; 200k rows in 0.94 stest_selection_never_computes_attributionmonkeypatches it to raiseOn the 4k-row fixture a full run (30 configurations, K = 3) takes about 4 s.
Also
require_selection_hygiene=True(also--require-selection-hygieneon the CLI). It reads what the block records and cannot verify it against data it does not have.sweep_whiteboxwarns whenalpha_gridis not passed (its default is the falsified assumption) and whenX_valis used both to choose and to report. Existing callers keep working.tests/test_validate.pynow expects check 11 to skip on artifacts without provenance.from compileml import compile_selectedis lazy, soimport compilemlstays free of NumPy and scikit-learn.Verification
26 new tests in
tests/test_batch.pyandtests/test_select.py: partition properties (disjoint and covering, out-of-time, group, explicit overlap), duplicates and row-id overlap, the prefix trick, the ceiling fit budget, the tie rule and its override by a clearly better configuration, the floor gate, no-ceiling behaviour, report-once with the hash covering the report, nested fallback, check 11 on a good artifact and on tampered provenance, both sweep warnings, the EPS warning, and the lazy import.ruff,black, fullpytestandmkdocs build --strictpass;mypy(advisory) shows no errors in the new or changed modules.Not in this PR
Docs, the tuning guide, the FAQ entry, the README quick example and roadmap, the API reference entries, and the benchmark rerun (PR 2). The logit latent (0.10.0, gated on #76). Any API rename.