Repository navigation
Closed
Labels
extension-modulesC modules in the Modules dirC modules in the Modules dirperformancePerformance or resource usagePerformance or resource usagetopic-IOtype-featureA feature request or enhancementA feature request or enhancement
Description
Feature or enhancement
BufferedRWPair inherits IOBase.readline(), which calls peek(1) to find the newline, then read(n) to consume the line. In C, each peek copies all unread bytes, up to 128 KiB by default, so reading short lines copy the same bytes over and over.
The simple idea is to relay readline() to the pair's BufferedReader.
| Call and line size | main | patch | speedup |
|---|---|---|---|
readline(), 16 B lines |
349 ns | 35.1 ns | 9.93x |
readline(), 80 B lines |
700 ns | 36.2 ns | 19.33x |
readline(), 1 KiB lines |
1.26 us | 95.2 ns | 13.24x |
readline(), 16 KiB lines |
6.23 us | 932 ns | 6.68x |
readline(), 64 KiB lines |
21.5 us | 3.18 us | 6.77x |
readline(65537), 16 B lines |
340 ns | 37.8 ns | 9.01x |
readline(65537), 80 B lines |
716 ns | 39.2 ns | 18.26x |
readline(65537), 1 KiB lines |
1.27 us | 97.8 ns | 13.03x |
readline(65537), 16 KiB lines |
6.54 us | 938 ns | 6.98x |
readline(65537), 64 KiB lines |
23.2 us | 3.23 us | 7.17x |
readline(), 16 B lines, 256 B buffer |
122 ns | 37.4 ns | 3.25x |
readline(), 80 B lines, 256 B buffer |
176 ns | 57.2 ns | 3.08x |
There are real users:
- Swift's memcached client (
readline()) - ptyprocess (
readline()) - sshuttle (
readline()) - viztracer (
readline()) - Sapling (
readline(1024))
BytesIO benchmark script (pyperf)
Ran python bench_rwpair_readline.py -p 10 with each build.
import io
import time
import pyperf
SIZES = ((16, "16 B"), (80, "80 B"), (1024, "1 KiB"),
(16384, "16 KiB"), (65536, "64 KiB"))
LIMIT = 65537
def rwpair(data, bufsize=io.DEFAULT_BUFFER_SIZE):
return io.BufferedRWPair(io.BytesIO(data), io.BytesIO(), bufsize)
def small_rwpair(data):
return rwpair(data, 256)
def reader(data):
return io.BufferedReader(io.BytesIO(data))
def bench(loops, make, data, count, args):
total = 0.0
for _ in range(loops):
readline = make(data).readline
t0 = time.perf_counter()
for _ in range(count):
readline(*args)
total += time.perf_counter() - t0
return total
def add(runner, name, make, size, args=()):
count = min(2048, max(16, (1 << 20) // size))
data = (b"x" * (size - 1) + b"\n") * count
runner.bench_time_func(name, bench, make, data, count, args,
inner_loops=count)
runner = pyperf.Runner()
for size, label in SIZES:
add(runner, f"readline(), {label} lines", rwpair, size)
for size, label in SIZES:
add(runner, f"readline({LIMIT}), {label} lines", rwpair, size, (LIMIT,))
for size, label in SIZES[:2]:
add(runner, f"readline(), {label} lines, 256 B buffer", small_rwpair, size)
for size, label in SIZES[1::2]:
add(runner, f"BufferedReader.readline(), {label} lines (control)",
reader, size)Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
Related discussion of IOBase.readline() in #159505
Linked PRs
Metadata
Metadata
Assignees
Labels
extension-modulesC modules in the Modules dirC modules in the Modules dirperformancePerformance or resource usagePerformance or resource usagetopic-IOtype-featureA feature request or enhancementA feature request or enhancement
Activity
cmaloney commentedon Nov 18, 2026
This looks reasonable and should make the users, including
pdba little faster. I had been wanting to removeBufferedRWPairbut thesocket.makefile()return shape necessitates it and better to maintain the existing then deprecate + make something new. Are some other general BufferedRWPair issues I'll make a new/separate issue for.maurycy commentedon Nov 18, 2026
@cmaloney Thank you. Looking forward to it! Happy to help.
gh-162230: Relay `BufferedRWPair.readline()` to the reader (#162231)