Gate experiment: cross-fitted α sweep with event thinning on public data — decides 0.10.0 #76

The experiment that decides 0.10.0. Companion to #75 (0.9.0); the RFC text will be linked here once posted.

Question

Three things are unmeasured, and each has been decided by one dataset or by an experiment with a known flaw:

1. Where soft targets stop helping. The only boundary so far — soft targets winning below roughly 10 events per split — came from one synthetic sweep that fitted students to the teacher's in-sample probabilities. In-sample predictions partly restate the label, which dilutes the soft-target arm, so that boundary is a lower bound: cross-fitted, soft targets should win at higher event counts, not lower. Nobody has measured by how much.
2. Whether log-loss beats squared error. The best configuration on the 2.9M-row portfolio was a log-loss student (0.8552 vs 0.8518), and the artifact contract currently refuses it. A logit latent would also remove the clamp-induced ladder collapse measured in Linear base term in the artifact: compile a logistic base plus a residual whitebox #39. But a schema bump on one portfolio's evidence is not enough.
3. Linear base term in the artifact: compile a logistic base plus a residual whitebox #39's second dataset. A 37-coefficient WoE logistic regression beat the compiled whitebox on one portfolio. It needs a second.

Public data keeps every published number free of any lender's portfolio, and lets anyone rerun it.

Datasets

Three datasets, two domains, base rates from 5% to 22%. Anyone with Kaggle access can run the whole thing; the Taiwan panel alone runs offline.

Method

Per dataset:

• Partitions. Fit / Select / Report, stratified (nested CV for the 1,400-row fraud sample). This is a measurement, not a shipped artifact, so the single-read rule does not apply; every configuration is evaluated on the held-out partition with bootstrap intervals, and the write-up says so.
• Ceiling. A tuned booster, tuned once on Fit, frozen, refit K = 5 times for out-of-fold predictions. Record the budget.
• Floor. fit_reference (WoE logistic) inside Fit.
• Event thinning. Subsample positives in Fit to hit target EPS levels, EPS = positives_fit / (trees × (2^depth − 1)), for example {2, 5, 10, 20, 50, 100}, at depth 2 and a fixed tree count. Report EPS alongside every result.
• Targets. α ∈ {0, 0.25, 0.5, 0.75, 1} against cross-fitted ceiling predictions. Also run the in-sample variant once, to measure the dilution directly.
• Losses. Squared error via train_whitebox. Log-loss lives outside the library for now: GradientBoostingClassifier on row-duplicated data — each row twice, labels 1 and 0, weights t and 1 − t — which is exact cross-entropy against a fractional target with no new dependency. Because the duplicates double the sample counts, state leaf and split constraints in weight form (min_weight_fraction_leaf, and the split constraint likewise), and keep subsample=1.0 so the two halves of a row are never drawn apart. Compare at the whitebox level; quantization affects both losses alike and its error bound is recorded by build_artifact.
• Metric. Gini on the held-out partition with 200-resample bootstrap intervals; Brier reported alongside.

Deliverables

• a script under benchmarks/ that runs the whole design from a dataset name, seeded, and writes one tidy CSV of results (dataset, EPS, α, loss, trees, depth, Gini, CI, Brier, floor Gini, ceiling Gini)
• a results table posted to this issue and to Show and tell, with the ceiling budget stated
• the EPS boundary per dataset, cross-fitted and in-sample side by side, so the dilution is measured rather than argued
• the log-loss decision: a non-inferiority test of log-loss against squared error per dataset, with a stated margin (proposal: 0.005 Gini, adjustable) and a 95% bootstrap interval on the difference
• Linear base term in the artifact: compile a logistic base plus a residual whitebox #39's second and third datasets: floor Gini against whitebox Gini on each
• whether any interior α ever wins cross-fitted, which sets the 1.0 default grid

The README and results.json are not touched by this issue; 0.9.0 owns the benchmark.

Decision rule for 0.10.0

Proceed with the logit latent, loss="log_loss", soft-target cross-entropy and log-odds extraction if log-loss is non-inferior to squared error on at least two datasets. Strict superiority is not required, because the logit latent carries benefits on its own — no clamp, so the #39 collapse cannot occur; scorecard points in log-odds, the traditional convention; and a natural home for #39's WoE base. If log-loss is inferior beyond the margin on two datasets, 0.10.0 does not happen and #39 is rebuilt in probability space.

What a positive result then requires, for the record: a schema bump with a logit_tree_ensemble score type (older runtimes already refuse an unknown schema_version); signed display arithmetic in the SQL and COBOL exporters, which currently use the non-negative div_rha form; band edges that may be negative; and extract_trees accepting log-odds models for the logit schema only.

Scope

Medium, and mostly compute. The design is fixed above; the work is the script, the runs, and an honest write-up. Good for a contributor with Kaggle access and a free afternoon of machine time; the Taiwan panel alone is a useful first run and needs nothing but pip install compileml.