Repository navigation
Fix lost input when IgzipDecompressor gets more than 4 GiB and stops at max_length - #337
Merged
rhpvorderman merged 1 commit intoNov 20, 2026
Conversation
decompress_buf() passes the input to isal_inflate() in parts of at most UINT32_MAX bytes. When the output reached max_length, only the inner loop ended. The outer loop then set avail_in to the next part and dropped the bytes of the current part that isal_inflate() had not read. With more than 4 GiB of input in one call, later calls returned only part of the data. Leave both loops instead, so that the code after them adds avail_in back to avail_in_real.
Collaborator
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IgzipDecompressor.decompress()loses input when one call gets more than 4 GiB of data and the output reachesmax_length.decompress_buf()insrc/isal/igzip_libmodule.cpasses the input toisal_inflate()in parts of at mostUINT32_MAXbytes, becauseavail_inis a 32-bit field. When the output reachesmax_length, only the inner loop ends. The outer loop then setsavail_into the next part and drops the bytes of the current part thatisal_inflate()hasn't read. CPython's_bz2module.c, which this function is based on, addsavail_inback to the remaining length after each call.On
develop, with 4.5 GiB of incompressible data compressed to 4,833,307,213 bytes,decompress(compressed, 2**20)followed bydecompress(b"")returns 539,224,874 of the 4,831,838,208 bytes. TheneofisFalseandneeds_inputisTrue. The decompressor keeps only the 538,339,918 bytes after the first part.This PR leaves both loops when the output reaches
max_length, so the existingself->avail_in_real += self->state.avail_in;after the loops keeps the unread bytes. The other ways out of the loops don't need a change, becauseisal_inflate()has read all ofavail_inby then.The new
testDecompress4GMaxLengthtest needs about 20 GiB of memory, so it's skipped liketestDecompress4G. With the skip removed, it fails ondevelopand passes on this branch. The test suite passes on Linux with CPython 3.14, with ISA-L built from its assembly code and from its portable C code.flake8andmypypass. CI covers the other Python versions.CPython's internal
zlib._ZlibDecompressorhas the same loop and returns the same 539,224,874 bytes on CPython 3.14.4. Thegzipmodule passes it at most 128 KiB per call, sogzipisn't affected.Checklist