Skip to content

Fix lost input when IgzipDecompressor gets more than 4 GiB and stops at max_length - #337

Merged
rhpvorderman merged 1 commit into
pycompression:developfrom
ptweezy:decompressor-max-length-4gib
Nov 20, 2026
Merged

rhpvorderman merged 1 commit into
pycompression:developfrom
ptweezy:decompressor-max-length-4gib

Conversation

@ptweezy

@ptweezy ptweezy commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

IgzipDecompressor.decompress() loses input when one call gets more than 4 GiB of data and the output reaches max_length.

decompress_buf() in src/isal/igzip_libmodule.c passes the input to isal_inflate() in parts of at most UINT32_MAX bytes, because avail_in is a 32-bit field. When the output reaches max_length, only the inner loop ends. The outer loop then sets avail_in to the next part and drops the bytes of the current part that isal_inflate() hasn't read. CPython's _bz2module.c, which this function is based on, adds avail_in back to the remaining length after each call.

On develop, with 4.5 GiB of incompressible data compressed to 4,833,307,213 bytes, decompress(compressed, 2**20) followed by decompress(b"") returns 539,224,874 of the 4,831,838,208 bytes. Then eof is False and needs_input is True. The decompressor keeps only the 538,339,918 bytes after the first part.

This PR leaves both loops when the output reaches max_length, so the existing self->avail_in_real += self->state.avail_in; after the loops keeps the unread bytes. The other ways out of the loops don't need a change, because isal_inflate() has read all of avail_in by then.

The new testDecompress4GMaxLength test needs about 20 GiB of memory, so it's skipped like testDecompress4G. With the skip removed, it fails on develop and passes on this branch. The test suite passes on Linux with CPython 3.14, with ISA-L built from its assembly code and from its portable C code. flake8 and mypy pass. CI covers the other Python versions.

CPython's internal zlib._ZlibDecompressor has the same loop and returns the same 539,224,874 bytes on CPython 3.14.4. The gzip module passes it at most 128 KiB per call, so gzip isn't affected.

Checklist

  • Pull request details were added to CHANGELOG.rst
  • Documentation was updated (if needed)

decompress_buf() passes the input to isal_inflate() in parts of at most UINT32_MAX bytes. When the output reached max_length, only the inner loop ended. The outer loop then set avail_in to the next part and dropped the bytes of the current part that isal_inflate() had not read. With more than 4 GiB of input in one call, later calls returned only part of the data. Leave both loops instead, so that the code after them adds avail_in back to avail_in_real.
@rhpvorderman

Copy link
Copy Markdown
Collaborator

Thanks!

@rhpvorderman
rhpvorderman merged commit fa857d3 into pycompression:develop Nov 20, 2026
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants