Skip to content

gh-129813: Use PyBytesWriter for strided mmap slices - #162257

Merged
vstinner merged 1 commit into
python:mainfrom
maurycy:mmap-pybyteswriter-slices
Nov 19, 2026
Merged

vstinner merged 1 commit into
python:mainfrom
maurycy:mmap-pybyteswriter-slices

Conversation

@maurycy

@maurycy maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Write large strided slices directly to the result, saving one alloc + copy. The PR extends #138831 to slices such as m[::2].

This is mostly a simplification but it's good on the memory front:

Slice Result size Peak before (bytes) Peak after (bytes)
m[::2] 4 KiB 8,225 4,129
m[::2] 8 MiB 16,777,249 8,388,641
m[::-1] 8 MiB 16,777,249 8,388,641
m[::1] (control) 8 MiB 8,388,641 8,388,641

No regression on the performance front:

Slice Result size Round 1 baseline Round 1 patched Round 2 baseline Round 2 patched
m[::2] 32 B 31.3 ns 30.4 ns 31.7 ns 30.5 ns
m[::2] 4 KiB 1.14 µs 1.07 µs 1.15 µs 1.12 µs
m[::2] 8 MiB 2.34 ms 2.10 ms 2.38 ms 2.12 ms
m[::-1] 8 MiB 2.30 ms 2.15 ms 2.28 ms 2.08 ms
m[::1] (control) 8 MiB 123 µs 126 µs 118 µs 159 µs

(both on macOS arm64)

Reproduce memory measurements
import mmap
import tracemalloc


def measure(size, step):
    with mmap.mmap(-1, size * abs(step)) as m:
        key = slice(None, None, step)
        tracemalloc.start()
        result = m[key]
        current, peak = tracemalloc.get_traced_memory()
        tracemalloc.stop()
        assert len(result) == size
        print(size, step, current, peak)


for size, step in [(4096, 2), (8 << 20, 2), (8 << 20, -1), (8 << 20, 1)]:
    measure(size, step)
pyperf
import mmap

import pyperf


runner = pyperf.Runner()
runner.metadata["mmap_module"] = mmap.__file__

for size, step in ((32, 2), (4096, 2), (8 << 20, 2), (8 << 20, -1), (8 << 20, 1)):
    length = size * abs(step)
    with mmap.mmap(-1, length) as mapping:
        mapping[:] = b"x" * length
        runner.timeit(
            f"mmap[::{step}], {size} output bytes",
            stmt=f"mapping[::{step}]",
            globals={"mapping": mapping},
        )

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Nov 19, 2026
@maurycy

maurycy commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

cc @vstinner Found while checking after the remote_debugging PyBytesWriter

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Straightforward usage of the PyBytesWriter API.

@vstinner

Copy link
Copy Markdown
Member

This is mostly a simplification but it's good on the memory front

Hey, it's not only a simplification, it's also a nice optimization!

Reducing the peak memory usage by a factor of 2 is great! Also, the slice operation is up to 1.11x faster than before, which is also nice to have.

@vstinner
vstinner merged commit 9f205da into python:main Nov 19, 2026
61 of 64 checks passed
@maurycy
maurycy deleted the mmap-pybyteswriter-slices branch November 19, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip news type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants