Skip to content

Select the target and the configuration on data the report never sees (compile_selected) - #77

Merged
orgoca merged 1 commit into
mainfrom
feat/compile-selected
Sep 27, 2026
Merged

orgoca merged 1 commit into
mainfrom
feat/compile-selected

Conversation

@orgoca

@orgoca orgoca commented Sep 27, 2026

Copy link
Copy Markdown
Owner

The code half of #75. The docs, README and benchmark rerun follow in a second PR, because they depend on the number this produces.

from compileml import compile_selected

result = compile_selected(X, y, ceiling=factory, reference="woe")
result.selection_curve   # one row per configuration, Select-partition figures
result.artifact          # refit on Fit ∪ Select, carries provenance
report = result.report() # Report, read once; a second call raises

What it does

  • Partitions. make_partitions: stratified 60/20/20; date= makes Report the latest slice; group= keeps one applicant in one partition; explicit index dicts are checked for overlap. row_id= overlap is a hard failure. Content-identical rows across partitions are counted and fail only above duplicate_threshold (0.01), since they are legitimate in credit data.
  • Ceiling. The factory is called once, its configuration frozen (a search's best_estimator_, or the model itself), and refit K times for out-of-fold soft targets. ceiling_oof= / ceiling_scores= for users who only have predictions; provenance then says user_asserted. ceiling_budget= is recorded verbatim so retention is comparable across artifacts.
  • Grid. α × trees × depth × monotone, endpoints always in the α grid, α forced to 1 with no ceiling. Trees are free: one fit per (α, depth, monotone) at the largest count, extracted once and truncated, verified equal to a separate fit. Each prefix is quantized, calibrated and banded on Fit and scored on Select through the batch scorer.
  • Selection. One-SE-simplest: bootstrap the best configuration's Select metric for its SE; everything within tie_se SEs ties; among ties prefer α = 1, then fewer trees, then lower depth. Recorded as a governance prior. selection_metric="brier" is opt-in.
  • Floor gate. Below the floor on Select: a result with no artifact, unless allow_below_floor=True.
  • Final refit on Fit ∪ Select; out-of-fold targets regenerated over the union.
  • Report. Gini, KS, Brier for artifact, ceiling and floor; retention and floor ratio at equal weight with bootstrap intervals; written into metadata.provenance.report and the artifact rehashed. A second call raises. This is hygiene, not proof, and the docstring says so.
  • Nested CV replaces the fixed Select below min_select_events (300), with a warning. In nested mode the factory is called once per outer fold and once for the final refit, since correctness matters more than cost on data that small.
  • Provenance: target, selection rule and SE, partition method, row counts and content hashes, duplicate count, soft-target cross-fitting, ceiling family, params hash, budget and fit counts, floor, eps_min, and the report.

Compute criteria from #75, each met

Criterion How
histogram backend for search and refit train_whitebox(backend="hist"), new and explicit; the default is unchanged
trees axis by truncation, never refitting extract_trees passes an ExtractedModel through; test_prefix_of_a_fit_is_the_smaller_fit
ceiling tuned once, refit K times frozen test_ceiling_is_tuned_once_and_refit_frozen counts: 1 factory call, 1 + K + 1 fits (+ K when α < 1 is chosen)
batch scorer, bit-identical to decide() compileml.batch.score_batch; 0 mismatches across linear/step/no calibration, float32, and missing values; 200k rows in 0.94 s
attribution never computed during selection test_selection_never_computes_attribution monkeypatches it to raise

On the 4k-row fixture a full run (30 configurations, K = 3) takes about 4 s.

Also

  • Check 11, selection hygiene, advisory unless require_selection_hygiene=True (also --require-selection-hygiene on the CLI). It reads what the block records and cannot verify it against data it does not have.
  • sweep_whitebox warns when alpha_grid is not passed (its default is the falsified assumption) and when X_val is used both to choose and to report. Existing callers keep working. tests/test_validate.py now expects check 11 to skip on artifacts without provenance.
  • from compileml import compile_selected is lazy, so import compileml stays free of NumPy and scikit-learn.

Verification

26 new tests in tests/test_batch.py and tests/test_select.py: partition properties (disjoint and covering, out-of-time, group, explicit overlap), duplicates and row-id overlap, the prefix trick, the ceiling fit budget, the tie rule and its override by a clearly better configuration, the floor gate, no-ceiling behaviour, report-once with the hash covering the report, nested fallback, check 11 on a good artifact and on tampered provenance, both sweep warnings, the EPS warning, and the lazy import. ruff, black, full pytest and mkdocs build --strict pass; mypy (advisory) shows no errors in the new or changed modules.

Not in this PR

Docs, the tuning guide, the FAQ entry, the README quick example and roadmap, the API reference entries, and the benchmark rerun (PR 2). The logit latent (0.10.0, gated on #76). Any API rename.

compile_selected implements the protocol of #75. Rows are split into Fit,
Select and Report: stratified, out-of-time by date, or by group so one
applicant lands in one partition. The ceiling (the teacher) is tuned once
by the user's factory, frozen, and refit K times for cross-fitted soft
targets; one search plus K fits, never K searches. Every (alpha, trees,
depth, monotone) configuration is compiled on Fit and scored on Select as
an integer artifact, so quantization and banding are priced too. The
simplest configuration statistically tied with the best wins, preferring
alpha = 1 as a governance prior recorded as the tie rule; the floor gates;
the winner is refit on Fit union Select; result.report() reads Report once
and writes the figures into a hash-covered provenance block, so what ships
carries its own reported cost. Below a configurable event count in Select,
nested cross-validation takes over with a warning.

Whether the whitebox learns from the ceiling's probabilities or from the
labels is now a selected parameter. Every step is also a plain function.

compileml.batch.score_batch is the runtime's score path over a matrix in
NumPy, asserted bit-identical to decide() on every row across calibration
modes, input precision and missing values; 200,000 rows in under a second.
Selection ranks compiled artifacts with it and never computes attribution.

The trees axis is free: extract_trees passes an ExtractedModel through, so
one fit at the largest tree count is extracted once and compiled from
prefixes, verified equal to a separate fit for both backends.
train_whitebox(backend="hist") selects the histogram backend explicitly,
measured at 1-2 s where the classic backend took 18 s at 30k rows and 293 s
at 300k; compile_selected uses it throughout, and the default is unchanged.

Validation check 11, selection hygiene, reads the provenance block and lists
broken protocol: soft targets not cross-fitted, partitions that overlap,
Report read more than once. Advisory unless required, following check 10.
sweep_whitebox warns when alpha_grid is not passed and when one holdout is
used both to choose and to report.

Part of #75; the documentation and the benchmark rerun follow separately.
@orgoca
orgoca merged commit d0605d2 into main Sep 27, 2026
20 checks passed
@orgoca
orgoca deleted the feat/compile-selected branch September 27, 2026 16:47
@orgoca orgoca mentioned this pull request Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant