Skip to content

gh-162153: Fix zlib._ZlibDecompressor dropping input over 4 GiB when max_length is reached - #162228

Open
Aathi-27 wants to merge 4 commits into
python:mainfrom
Aathi-27:gh-162153-zlib-maxlen
Open

Aathi-27 wants to merge 4 commits into
python:mainfrom
Aathi-27:gh-162153-zlib-maxlen

Conversation

@Aathi-27

Copy link
Copy Markdown

When a single zlib._ZlibDecompressor.decompress(data, max_length) call receives more than UINT_MAX bytes of input and reaches the output limit, decompress_buf() re-ran its outer loop and arrange_input_buffer() overwrote zst.avail_in without crediting the unconsumed remainder to avail_in_real. Only the last chunk's remainder was added back after the loop, so up to ~4 GiB of input was silently dropped: eof stayed false, needs_input became true, and feeding the rest of the stream continued at the wrong position (gh-162153).

Once the output limit is reached, no more input is needed (no inflate() runs after it), so the outer loop now stops there and the existing final add-back accounts for the unconsumed remainder. _bz2module.c handles the same structure correctly via its per-call accounting.

Testing:

  • New bigmem testDecompressMaxLength4G in Lib/test/test_zlib.py: stored-block compressed input above 4 GiB drained with max_length=65536, asserting the full output length and eof. It runs at reduced (dry-run) scale in normal test runs and at full scale only on bigmem machines; standard CI does not run bigmem tests.
  • test_zlib passes on a pydebug build of main (91 tests run, 2 environmental skips: test_big_buffer needs 4 GB RAM, test_zlibng_version needs zlib-ng), plus test_gzip, test_zipfile, test_tarfile, test_shutil and test_bz2 (1,747 tests run).
  • The reporter's reproducer was run on main before the fix and confirmed the loss (4,294,573,779 bytes dropped).
  • Full-scale post-fix verification is still outstanding: the verification environment available for this work had ~1.6 GiB RAM, and with correct accounting the decompressor must buffer ~4.3 GiB of unconsumed input, so the full-scale reproducer could not run there (MemoryError, an observed resource constraint). The fix was additionally verified at reduced scale by temporarily lowering the internal input-chunk cap and running the same scenario: the unfixed code loses input and the fixed code returns it all. A full-scale run of the new bigmem test (or the reporter's reproducer) on a machine with ~11 GiB+ RAM would close this gap.

Fixes an issue where input exceeding 4 GiB was not handled correctly, leading to incorrect stream position during decompression.
@python-cla-bot

python-cla-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@bedevere-app

bedevere-app Bot commented Nov 18, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant