fmt: bound line buffering for inputs without newlines by mattsu2020 · Pull Request #10602 · uutils/coreutils

mattsu2020 · 2026-01-30T23:50:37Z

Summary

fmt currently buffers an entire line before producing output, so newline-free streams such as /dev/zero can grow memory indefinitely. Bound each read to 1 MiB and end the formatted paragraph at that boundary, without performing another potentially blocking read.

Carry at most three trailing bytes into the next read to avoid splitting UTF-8 characters or mishandling CRLF.
Retain the prefix/skip-prefix formatting decision across chunks of the same logical line.
Emit unformatted chunks without adding newlines at chunk boundaries; normalize LF/CRLF and append the final newline at the logical line end or EOF.

Compatibility

Formatted logical lines exceeding the safety limit are intentionally split into bounded paragraphs. Their wrapping and indentation can differ from GNU fmt, which retains arbitrarily large finite lines. Unformatted lines retain their bytes apart from the existing line-ending normalization.

This bounds newline-free reads; it does not introduce a general size limit for paragraphs made of newline-terminated lines.

Validation

cargo test -p uu_fmt: 8 passed.
cargo test --no-default-features --features fmt --test tests test_fmt::: 41 passed, 2 existing tests ignored.
cargo clippy -p uu_fmt --tests -- -D warnings: passed, with an existing workspace warning about the unknown clippy::assert_is_empty lint.
Formatting checks on the three changed Rust files and git diff --check: passed.

Regression coverage includes stalled input at exactly 1 MiB, 2/3/4-byte UTF-8 characters across the limit, LF/CRLF, EOF at exact and repeated limits, and matching/nonmatching prefix and skip-prefix continuation.

Related: #10580

github-actions · 2026-01-31T00:00:18Z

GNU testsuite comparison:

Skipping an intermittent issue tests/shuf/shuf-reservoir (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/sort/sort-stale-thread-mem (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/basenc/bounded-memory is now passing!

github-actions · 2026-02-03T11:57:38Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/follow-name (passes in this run but fails in the 'main' branch)

github-actions · 2026-02-14T07:27:41Z

GNU testsuite comparison:

GNU test failed: tests/pr/bounded-memory. tests/pr/bounded-memory is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/symlink (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/factor/t33 is no longer failing!
Congrats! The gnu test tests/factor/t34 is no longer failing!
Note: The gnu test tests/tail/pipe-f is now being skipped but was previously passing.

codspeed · 2026-02-14T07:32:15Z

Merging this PR will improve performance by 3.19%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 322 untouched benchmarks
⏩ 46 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`du_max_depth_balanced_tree[(6, 4, 10)]`	26.1 ms	25.3 ms	+3.19%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing mattsu2020:fmt_devzero (5ee4496) with main (5de3062)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-02-14T12:11:43Z

GNU testsuite comparison:

GNU test failed: tests/factor/t10. tests/factor/t10 is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/factor/t33. tests/factor/t33 is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/factor/t34. tests/factor/t34 is passing on 'main'. Maybe you have to rebase?
Skipping an intermittent issue tests/tty/tty-eof (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/factor/t28 is no longer failing!
Congrats! The gnu test tests/cut/cut-huge-range is now passing!
Congrats! The gnu test tests/rm/many-dir-entries-vs-OOM is now passing!

github-actions · 2026-02-18T08:08:56Z

GNU testsuite comparison:

GNU test failed: tests/timeout/timeout-group. tests/timeout/timeout-group is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/symlink (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/basenc/bounded-memory is now being skipped but was previously passing.

github-actions · 2026-02-18T08:25:39Z

GNU testsuite comparison:

GNU test failed: tests/date/date-locale-hour. tests/date/date-locale-hour is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/cut/bounded-memory is no longer failing!
Note: The gnu test tests/printf/printf-surprise is now being skipped but was previously passing.
Note: The gnu test tests/rm/many-dir-entries-vs-OOM is now being skipped but was previously passing.

github-actions · 2026-02-28T13:27:56Z

GNU testsuite comparison:

Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/symlink (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/timeout/timeout-group is no longer failing!
Note: The gnu test tests/seq/seq-epipe is now being skipped but was previously passing.
Congrats! The gnu test tests/tail/tail-n0f is now passing!

github-actions · 2026-03-24T10:42:14Z

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/csplit/csplit-heap is now passing!

sylvestre

too much added complexity for a rare case
please simplify it

github-actions · 2026-03-24T11:27:52Z

GNU testsuite comparison:

Skip an intermittent issue tests/date/resolution (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/cut/bounded-memory (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/tail/tail-n0f is now being skipped but was previously passing.

mattsu2020 · 2026-03-24T11:43:50Z

too much added complexity for a rare case please simplify it

I tried to keep it as simple as possible

github-actions · 2026-06-05T13:14:27Z

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/misc/tty-eof (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/tail-n0f (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/tail/pipe-f is now passing!

mattsu2020 · 2026-06-19T04:13:18Z

I simplified the implementation quite a bit.

Limit each read to 1 MiB and end formatted paragraphs at the safety boundary so newline-free streams can produce output without waiting for EOF or another read. Carry incomplete UTF-8 characters and a trailing CR into the next read. Keep the formatting decision for each logical line, and let pass-through chunks supply a newline only at the logical line end. Cover stalled input, UTF-8 splits, LF/CRLF boundaries, EOF, and prefix handling in unit and integration tests.

mattsu2020 force-pushed the fmt_devzero branch from 4e26790 to 4a1f5b0 Compare February 18, 2026 08:17

sylvestre requested changes Mar 24, 2026

View reviewed changes

mattsu2020 marked this pull request as draft March 24, 2026 11:10

mattsu2020 requested a review from sylvestre March 24, 2026 11:43

mattsu2020 marked this pull request as ready for review March 24, 2026 11:43

mattsu2020 force-pushed the fmt_devzero branch 2 times, most recently from 1405660 to c3bf2d5 Compare June 19, 2026 02:53

mattsu2020 marked this pull request as draft June 19, 2026 03:03

mattsu2020 force-pushed the fmt_devzero branch 2 times, most recently from 6b5bc2e to 5ee4496 Compare June 19, 2026 03:17

mattsu2020 changed the title ~~fmt: fix fmt /dev/zero panic~~ fmt: bound line buffering for inputs without newlines Jun 19, 2026

mattsu2020 marked this pull request as ready for review June 19, 2026 04:13

mattsu2020 force-pushed the fmt_devzero branch 2 times, most recently from a7fde05 to ae8fd88 Compare October 12, 2026 11:41

mattsu2020 force-pushed the fmt_devzero branch from ae8fd88 to bb3cd08 Compare October 12, 2026 11:50

Conversation

mattsu2020 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Compatibility

Validation

Uh oh!

github-actions Bot commented Jan 31, 2026

Uh oh!

github-actions Bot commented Feb 3, 2026

Uh oh!

github-actions Bot commented Feb 14, 2026

Uh oh!

codspeed Bot commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 3.19%

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Feb 14, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 28, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

sylvestre left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

mattsu2020 commented Mar 24, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattsu2020 commented Jun 19, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mattsu2020 commented Jan 30, 2026 •

edited

Loading

codspeed Bot commented Feb 14, 2026 •

edited

Loading

github-actions Bot commented Jun 5, 2026 •

edited

Loading