Skip to content

gh-129813: Use PyBytesWriter in binascii - #162259

Merged
vstinner merged 1 commit into
python:mainfrom
maurycy:binascii-qp-pybyteswriter
Nov 19, 2026
Merged

vstinner merged 1 commit into
python:mainfrom
maurycy:binascii-qp-pybyteswriter

Conversation

@maurycy

@maurycy maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Use PyBytesWriter in binascii.b2a_qp() and a2b_qp(), spares a temp buffer and a copy. This extends #138825.

This is mostly a simplification but there's memory improvement:

Function and input Result size Baseline peak Patched peak
b2a_qp text 1,048,576 2,097,185 1,048,609
b2a_qp binary 3,229,614 6,459,261 3,229,647
a2b_qp text 1,048,576 2,359,329 1,310,753
a2b_qp hex escapes 349,525 1,398,133 1,048,608
a2b_qp soft breaks 0 1,048,575 1,048,608

And pyperf improvement:

Function and input Baseline 1 Patched 1 Baseline 2 Patched 2
b2a_qp 12 B 76.6 ns 74.1 ns 77 ns 75.8 ns
b2a_qp 255 B 1.21 µs 1.22 µs 1.23 µs 1.24 µs
b2a_qp text 4.73 ms 4.73 ms 4.72 ms 4.76 ms
b2a_qp binary 2 ms 1.85 ms 2 ms 1.74 ms
a2b_qp 12 B 34.4 ns 36.5 ns 35.1 ns 32.4 ns
a2b_qp text 886 µs 887 µs 883 µs 898 µs
a2b_qp hex escapes 447 µs 435 µs 444 µs 439 µs
a2b_qp soft breaks 419 µs 420 µs 416 µs 424 µs
a2b_hex control 125 ns 128 ns 127 ns 127 ns

(PGO+LTO, macOS arm64)

memory measure
import binascii
import tracemalloc

for func, case, data in (
    (binascii.b2a_qp, "text", b"abc def\n" * 131072),
    (binascii.b2a_qp, "binary", b"\xff" * 1048576),
    (binascii.a2b_qp, "text", b"abc=20def\n" * 131072),
    (binascii.a2b_qp, "hex escapes", b"=FF" * 349525),
    (binascii.a2b_qp, "soft breaks", b"=\r\n" * 349525),
):
    tracemalloc.start()
    result = func(data)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    print(func.__name__, case, len(result), peak)
    del result
pyperf benchmark
import binascii
import pyperf

runner = pyperf.Runner()
for func, case, data in (
    (binascii.b2a_qp, "12 B", b"Hello world!"),
    (binascii.b2a_qp, "255 B", b"a" * 255),
    (binascii.b2a_qp, "text", b"abc def\n" * 131072),
    (binascii.b2a_qp, "binary", b"\xff" * 1048576),
    (binascii.a2b_qp, "12 B", b"Hello world!"),
    (binascii.a2b_qp, "text", b"abc=20def\n" * 131072),
    (binascii.a2b_qp, "hex escapes", b"=FF" * 349525),
    (binascii.a2b_qp, "soft breaks", b"=\r\n" * 349525),
    (binascii.a2b_hex, "control", b"ab" * 128),
):
    runner.timeit(f"{func.__name__} {case}", "func(data)",
                  globals={"func": func, "data": data})

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Nov 19, 2026
@maurycy

maurycy commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

cc @vstinner Another found after checking the remote_debugging PyBytesWriter

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice usage of PyBytesWriter_FinishWithSize() to truncate the overallocated bytes object.

The old code initialized bytes to zero (PyMem_Calloc()) which is not needed and had to copy bytes (PyMem => PyBytes). I expect that the new code has less work to do and so can be a little bit faster.

Reducing the peak memory footprint by a factor of 2 is great! Some operations are also up to 1.08x faster according to your benchmark.

@vstinner
vstinner enabled auto-merge (squash) November 19, 2026 10:01
@vstinner
vstinner merged commit 03e7a88 into python:main Nov 19, 2026
59 of 64 checks passed
@maurycy
maurycy deleted the binascii-qp-pybyteswriter branch November 19, 2026 10:05
@maurycy

maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@vstinner Thank you :) for taking a look and merging. Truth to be told, at some point I realized it's better to underhype any pyperf improvements, and in the series - the case was strong for simplicity and better memory usage (thank you for PyBytes*()!). Sooo much depends on a specific PGO+LTO layout (good example) that I always think that anything between 5-10% might be a noise, even if there are very solid grounds like less allocs etc.

@vstinner

Copy link
Copy Markdown
Member

at some point I realized it's better to underhype any pyperf improvements

Yeah, that's why I added "skip news", I'm sure that it's worth it document such optimization. While the lower peak memory usage is great, the CPU time is faster but no so much faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip news type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants