Skip to content

io: BufferedRWPair.readline() copies the unread buffer for every line #162230

Closed

Description

@maurycy

Feature or enhancement

BufferedRWPair inherits IOBase.readline(), which calls peek(1) to find the newline, then read(n) to consume the line. In C, each peek copies all unread bytes, up to 128 KiB by default, so reading short lines copy the same bytes over and over.

The simple idea is to relay readline() to the pair's BufferedReader.

Call and line size main patch speedup
readline(), 16 B lines 349 ns 35.1 ns 9.93x
readline(), 80 B lines 700 ns 36.2 ns 19.33x
readline(), 1 KiB lines 1.26 us 95.2 ns 13.24x
readline(), 16 KiB lines 6.23 us 932 ns 6.68x
readline(), 64 KiB lines 21.5 us 3.18 us 6.77x
readline(65537), 16 B lines 340 ns 37.8 ns 9.01x
readline(65537), 80 B lines 716 ns 39.2 ns 18.26x
readline(65537), 1 KiB lines 1.27 us 97.8 ns 13.03x
readline(65537), 16 KiB lines 6.54 us 938 ns 6.98x
readline(65537), 64 KiB lines 23.2 us 3.23 us 7.17x
readline(), 16 B lines, 256 B buffer 122 ns 37.4 ns 3.25x
readline(), 80 B lines, 256 B buffer 176 ns 57.2 ns 3.08x

There are real users:

BytesIO benchmark script (pyperf)

Ran python bench_rwpair_readline.py -p 10 with each build.

import io
import time

import pyperf

SIZES = ((16, "16 B"), (80, "80 B"), (1024, "1 KiB"),
         (16384, "16 KiB"), (65536, "64 KiB"))
LIMIT = 65537


def rwpair(data, bufsize=io.DEFAULT_BUFFER_SIZE):
    return io.BufferedRWPair(io.BytesIO(data), io.BytesIO(), bufsize)


def small_rwpair(data):
    return rwpair(data, 256)


def reader(data):
    return io.BufferedReader(io.BytesIO(data))


def bench(loops, make, data, count, args):
    total = 0.0
    for _ in range(loops):
        readline = make(data).readline
        t0 = time.perf_counter()
        for _ in range(count):
            readline(*args)
        total += time.perf_counter() - t0
    return total


def add(runner, name, make, size, args=()):
    count = min(2048, max(16, (1 << 20) // size))
    data = (b"x" * (size - 1) + b"\n") * count
    runner.bench_time_func(name, bench, make, data, count, args,
                           inner_loops=count)


runner = pyperf.Runner()
for size, label in SIZES:
    add(runner, f"readline(), {label} lines", rwpair, size)
for size, label in SIZES:
    add(runner, f"readline({LIMIT}), {label} lines", rwpair, size, (LIMIT,))
for size, label in SIZES[:2]:
    add(runner, f"readline(), {label} lines, 256 B buffer", small_rwpair, size)
for size, label in SIZES[1::2]:
    add(runner, f"BufferedReader.readline(), {label} lines (control)",
        reader, size)

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

Related discussion of IOBase.readline() in #159505

Linked PRs

Activity

  1. cmaloney commented on Nov 18, 2026

    @cmaloney
    Contributor

    This looks reasonable and should make the users, including pdb a little faster. I had been wanting to remove BufferedRWPairbut the socket.makefile() return shape necessitates it and better to maintain the existing then deprecate + make something new. Are some other general BufferedRWPair issues I'll make a new/separate issue for.

  2. maurycy commented on Nov 18, 2026

    @maurycy
    ContributorAuthor

    @cmaloney Thank you. Looking forward to it! Happy to help.

  3. added a commit that references this issue on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions