[bercedev]

Description

Fixes #1926

_binomial_cdf started its pmf recurrence from (1 - p) ** n. Once n * log(1 - p) < -745, that term is 0.0 in float64, so every later term and the CDF were 0.0 as well. The Clopper-Pearson bisection then followed a wrong CDF, and success_rate_ci gave intervals that were far too wide or excluded the observed rate.

As the issue suggests, _binomial_cdf now works in log space. Each term's logarithm is lgamma(n + 1) - lgamma(i + 1) - lgamma(n - i + 1) + i log p + (n - i) log1p(-p). The terms are summed with math.fsum as exp(log_term - peak) and scaled back by exp(peak), clamped to at most 1. The early returns and the bisection are unchanged, and the docstring now explains the underflow.

I checked the new CDF against an exact Fraction sum for n ∈ {10, 200, 2000} and p ∈ {0.01, 0.5, 0.97, 0.999}. The worst relative error was 1.5e-12. _binomial_cdf(1000, 2000, 0.5) now gives 0.50892, where it used to give 0.0. One trade-off: each term costs three lgamma calls, so a 2000-trial interval takes about 0.3 s on my machine.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (the _binomial_cdf docstring)
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

Tests (tests/unit/utilities/test_statistics.py):

• _binomial_cdf against an exact rational sum (math.comb and Fraction) at (1000, 2000, 0.5), (190, 200, 0.97), (1995, 2000, 0.999) and (5, 2000, 0.01), with rel=1e-9;
• success_rate_ci against the issue's scipy reference bounds for 199/200, 297/300, 1000/2000 and 1990/2000 (four places, abs=1e-4), and the interval contains the point estimate;
• the all-success lower bound at n = 200 and 1000 against the closed form 0.025 ** (1 / n).

On dev (c5affed) 8 of the new cases fail. The two CDF cases that pass there don't underflow. With the change, pytest tests/unit/utilities/test_statistics.py transformer_lens/utilities/statistics.py gives 40 passed, including the module doctests and the 0/6 → [0.0, 0.459] example. black --check and isort --check-only are clean on both files. I ran all of this on Python 3.12 with CPU torch, offline in a container.

Note: I used Claude Code while working on this.

[jlarson4]

@bercedev Thanks for putting this together! I will rerun any failing CI tests and give this a review once the CI run finishes. We had a lot of PRs open at once so that can often trigger rate limit errors. I will let you know when my full review is ready.

[jlarson4]

Would you please comment on #1926 so I can assign you?

[jlarson4]

This looks good @bercedev, no comments from me, merging now!