Repository navigation
Conversation
The freelist was hardcoded to only allow one digit ints. `ob_digit` is the last member and `sizeof(PyLongObject)` actually has space for two on 64 bit machines. Refactor to add a MAX_DIGITS_FOR_FREELIST which computes what fits and use the freelist for all PyLongObject. For 64 bit platforms that means 2-digit `int` now use the freelist while taking no extra space. Each digit stores 30 bits so this still doesn't reach platform integer length on 64 bit (2 * 30 < 64). Adding another digit would change the allocation class unfortunately so need other approaches. PGO+LTO, upstream/main b90572a vs this change, mean ns: | statement | main | branch | delta | |-----------------------------------|-------:|-------:|--------:| | `s + s` (1 digit + 1 digit -> 2) | 17.2 | 13.7 | -3.4 | | `s * t` (1 digit * 1 digit -> 2) | 16.8 | 13.6 | -3.2 | | `a - b` (2 digits) | 16.9 | 14.0 | -2.8 | | `a // 7` | 20.8 | 17.7 | -3.1 | | `a & b` | 16.1 | 13.1 | -2.9 | | `a >> 3` | 17.6 | 14.6 | -3.0 | | `id(m)` | 17.2 | 14.5 | -2.7 | | `int('1234567890123')` | 39.2 | 34.6 | -4.6 | | `time.time_ns()` | 35.1 | 30.8 | -4.4 | | `time.perf_counter_ns()` | 34.9 | 29.3 | -5.6 | | `sum(range(2**31, 2**31 + 1000))` | 9189.7 | 6125.9 | -3063.8 | | `str(5)` | 25.1 | 21.5 | -3.5 | | `str(123456789)` | 29.8 | 26.8 | -3.0 | | `f'{n}'` (1 digit) | 30.0 | 25.6 | -4.4 | | `'%d' % n` | 50.2 | 46.0 | -4.3 | | `b'%d' % n` | 38.2 | 35.3 | -2.9 | | `a + b` (allocates 3 digits) | 16.9 | 15.8 | -1.1 | | `a * 3` (allocates 3 digits) | 20.9 | 21.0 | +0.2 | | `a << 1` (allocates 3 digits) | 19.1 | 18.3 | -0.9 | | `str(2**62)` (3-digit scratch) | 35.0 | 35.2 | +0.2 | | `m + k` (1 digit, control) | 9.6 | 9.0 | -0.5 | | `c + d` (3 digits, control) | 16.0 | 16.6 | +0.7 | | `str(10**40)` (control) | 113.6 | 113.0 | -0.6 | pyperformance, one pass per side: | Benchmark | main | branch | |------------------|:-------:|:---------------------:| | crypto_pyaes | 34.5 ms | 31.3 ms: 1.10x faster | | mako | 6.26 ms | 6.05 ms: 1.03x faster | | json_dumps | 4.33 ms | 4.21 ms: 1.03x faster | | pprint_safe_repr | 364 ms | 353 ms: 1.03x faster | | pprint_pformat | 739 ms | 721 ms: 1.02x faster | | deepcopy | 110 us | 109 us: 1.01x faster | | pathlib | 9.12 ms | 9.26 ms: 1.02x slower | | chameleon | 4.60 ms | 4.69 ms: 1.02x slower | | telco (control) | 3.49 ms | 3.38 ms: 1.03x faster | | pyflate (control) | 205 ms | 199 ms: 1.03x faster | | richards (control) | 21.1 ms | 20.8 ms: 1.01x faster | | spectral_norm (control) | 45.1 ms | 44.7 ms: 1.01x faster | | pidigits (control) | 109 ms | 110 ms: 1.01x slower | | tomli_loads (control) | not significant | | Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The case |
|
working on a fix for the c analyzer. Could also remove the static assert here, but I'd rather keep it. Parens with more than one comma should be supported. Separate issue and PR (Time to think about this isn't a bad thing / still early in 3.16 dev cycle) |
|
Opened gh-162182 for the c-analyzer failure |
| _Py_SetImmortal(self); | ||
| return; | ||
| } | ||
| if (_PyLong_IsCompact((PyLongObject *)self)) { |
There was a problem hiding this comment.
I think this should actually be ==. While it's unlikely on 64 bit someone appears with a 1-digit PyLongObject after this change keeping to only one size of possible object in the freelist seems good
There was a problem hiding this comment.
Having PyLong_DigitCount((PyLongObject *)self) < MAX_DIGITS_FOR_FREELIST should not indeed not happen (could happen for subclasses, but here we are in _PyLong_ExactDealloc).
Add an assert that _PyLong_DigitCount((PyLongObject *)self) >= MAX_DIGITS_FOR_FREELIST). (the check can then be either <= or ==
There was a problem hiding this comment.
There isn't a good way to get "capacity" unfortunately without adding some significant hooks to the allocator that I don't think are worth it for this safety.
Moved so in debug builds and with ASAN enabled we write to the required freelist capacity which should help.
The check stays <= because PyLongObject ndigits can be shrunk during usage, but we should always have sizeof(PyLongObject) min space.
memset the space which will result in python debug allocator and ASAN failures. There's no cheap way without a lot of new code to see exactly how much space was actually allocated. This makes too small of Objects getting in reasonably noisy
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The freelist was hardcoded to only allow one digit
int. Each digit is 32 bits and can hold 30 bits of data.ob_digitis the last member andsizeof(PyLongObject)actually has space for two on 64 bit machines.Refactor to add a MAX_DIGITS_FOR_FREELIST which computes at compile time what fits and use the freelist for all PyLongObject that fit. For 64 bit platforms that means 2-digit
intnow use the freelist while taking no extra space.Each digit stores 30 bits so this still doesn't reach platform integer length on 64 bit (2 * 30 < 64). Adding another digit would change the allocation class unfortunately so need other approaches.
PGO+LTO, upstream/main b90572a vs this change, mean ns:
s + s(1 digit + 1 digit -> 2)s * t(1 digit * 1 digit -> 2)a - b(2 digits)a // 7a & ba >> 3id(m)int('1234567890123')time.time_ns()time.perf_counter_ns()sum(range(2**31, 2**31 + 1000))str(5)str(123456789)f'{n}'(1 digit)'%d' % nb'%d' % na + b(allocates 3 digits)a * 3(allocates 3 digits)a << 1(allocates 3 digits)str(2**62)(3-digit scratch)m + k(1 digit, control)c + d(3 digits, control)str(10**40)(control)pyperformance, one pass per side:
intdigit count available in freelists #162150