- Fix
igzip_lib.IgzipDecompressor.decompress()losing input when one call gets more than 4 GiB of data and the output reachesmax_length. Later calls returned only part of the data, andneeds_inputbecameTruebefore the end of the stream. - Make
isal_zlib.Compress.flush()returnb""forZ_SYNC_FLUSHright after a sync or full flush, and forZ_FULL_FLUSHright after a full flush, as zlib does. It wrote an empty block of 6 bytes, or 117 bytes at level 0. On Linux, such a flush is now 20 times faster at levels 1 to 3. Flushing anigzipfile again without a write in between no longer grows the file and is 4 times faster. - Speed up
import isalby about 0.1 ms in programs that have not imported__future__yet. The version module that is generated when isal is built no longer imports__future__. Building isal now requires setuptools-scm 9.2 or later. - Fix
igzip_threadedfiles that stopped working when the thread that opened them ended, for example aThreadPoolExecutorthread aftershutdown(). Reads returned truncated data without an error, and writes waited forever. - Reduce the memory use of
igzip_threaded.openreaders. They now read the compressed input in pieces of 2 timesblock_sizeinstead of 8 times, which saves up to 6 MiB per open file with the defaultblock_sizeof 1 MiB. When reading from a pipe, the first data is now returned once 2 MiB of compressed data has arrived, instead of 8 MiB. - Lower the memory use of
igzip_threadedwriters on CPython. Threads that wait for their next block no longer keep their last blocks in memory. When data is written slower than the threads compress it, writing with 8 threads and the default 1 MiB blocks uses about 7 MiB less memory at its peak. - Fix
tell()and forwardseek()onigzip.IGzipFileobjects that read from an unseekable file object, such as a pipe or a socket. Versions 1.4.0 to 1.9.1 raisedio.UnsupportedOperationfor these calls. - Fix
python -m isal.igzipreplacing the input file with empty output and exiting with status 0 when the output file is the input file, as in-f -o data data. It exits with an error now. - Speed up
import isal.igzipandimport isal.igzip_threadedby not importingtyping. - Fix the gzip header that
igzip_threaded.openwrites whenthreadsis not 0. It had the XFL and OS bytes swapped, so thefilecommand reported the output as coming from VM/CMS or a FAT file system. - Fix pickling of
isal_zlib.error(igzip_lib.IsalError). An error raised in amultiprocessingworker process did not reach the parent process. Tracebacks now showisal.igzip_lib.IsalErrorinstead ofigzip_lib.IsalError.
- Fix BrokenPipeError when piping
python -m isal.igzipintohead,lessor similar programs.
By far most of the work on this release has been done by @ptweezy. Almost all bug fixes and improvements are their work.
Major changes:
- Updated included ISA-L library to 2.32.1.
- Python 3.10 and PyPy 3.10 are no longer supported. Added support for PyPy 3.12 and Python 3.15.
- Restore PyPy wheel builds.
- Fix builds on macOS with Apple Silicon, which used the portable C code of ISA-L rather than its optimized aarch64 assembly. On an Apple M4, decompression is 1.6 to 3 times faster, CRC-32 is 50 times faster, and Adler-32 is 7 times faster.
Optimizations and features:
isal_zlib.compressobjsupportsstrategy=Z_FIXEDat compression level 0. Level 0 then uses fixed Huffman codes instead of starting each block with a Huffman table of about 110 bytes. This shrinks small flushed writes, such as websocket messages: JSON messages of about 440 bytes, each followed byflush(Z_SYNC_FLUSH), compress to an average of 75 instead of 188 bytes. Blocks of a few KiB or more can be larger: about 5% for JSON and 10% for Python source code at 64 KiB. Other strategies, andZ_FIXEDat levels 1 to 3, still give a warning and use the default strategy.- Speed up
import isal.igzipandimport isal.igzip_threadedby importingargparseandshutilonly when the command line interface runs, and by dropping themultiprocessingimport. - Speed up reading lines from binary
igzipfiles through iteration,readlines(), andreadline(). On an Apple M4, reading FASTQ data is 1.13 times faster, and reading lines of a few bytes is 1.5 times faster. - Speed up each
write()call to binaryigzipfiles on CPython. On an AMD Ryzen 7 9800X3D with Python 3.14.8, writing FASTQ data or Python source code one line at a time is 1.1 times faster. - Speed up many small writes to
igzip.IGzipFileon Python 3.12 and later.IGzipFilecollects these writes in the write buffer ofgzip.GzipFileand compresses them in large chunks, which makes writing lines 1.4 times faster.
Bug fixes:
- Fix zlib streams compressed with a preset dictionary so they can be decoded by standard zlib, including builds linked against older ISA-L releases.
- Fix backward seeking in igzip files by discarding buffered compressed data when rewinding the underlying file.
- Fix a crash when
igzip.decompressgets an object that is not bytes-like, such as astr. It raises aTypeErrorinstead. - Fix a crash after creating
igzip_lib.IgzipDecompressorobjects with azdict. - Fix error handling when more than one interpreter imports
isal.isal_zliborisal.igzip_lib, for example, the main interpreter and a legacy subinterpreter thatPy_NewInterpreter()creates. Each import replaced the error classes that the C code raises, soexcept isal_zlib.error,except igzip_lib.error, andexcept gzip.BadGzipFilecaught errors only in the interpreter that imported the modules last. - Fix a hang in
isal_zlib.Compress: afterflush()raised an error for an unsupported mode, the next call on the same object never returned. - Fix a crash when you call
flush(Z_FINISH)twice on a level 0isal_zlib.Compressobject. - Fix
unused_dataofisal_zlib.Decompressandigzip_lib.IgzipDecompressor, which could contain bytes from outside the input buffer whenmax_lengthsplit the end of a stream. At the end of a stream,isal_zlib.Decompress.unconsumed_tailis empty, matching the CPython fix for gh-158169. - Fix
igzip_lib.IgzipDecompressor.needs_input, which could beTrueafterdecompress()returnedmax_lengthbytes while more output was still available. Readers that stop at the end of the file whenneeds_inputisTrue, such as CPython'sDecompressReader, raisedEOFErrorfor valid data. - Fix
isal_zlib.decompressobjwith awbitsof 40 to 47, which detects zlib or gzip data. It failed on gzip data when the firstdecompress()call got fewer than 2 bytes, for example, when the data arrived byte by byte. - Fix
isal_zlib.decompressobjwith awbitsof 40 to 47 after aflush()call that came before any data. It then failed on both zlib and gzip data. igzip_threaded.openraisesValueErrorfor ablock_sizeof about 3.6 GiB or more andTypeErrorfor acompresslevelthat is not an integer. Both raisedSystemErrorbefore.- Fix
igzip_threadedwriters at compression level 0, which raisedOverflowErrorfor data that does not compress, such as random bytes or already compressed files. At compression level 0,block_sizemust now be less than about 2 GiB. - Extraordinarily large gzip headers with either a very long comment or
filename will now cause an
OverflowErrorrather than use more and more memory. The size is limited to the internal buffer size which is currently set at 512 KiB. - Fix
igzip_threadedreaders and writers that waited for a polling timeout of up to 50 ms at the end of the stream or when closed. Writing and closing a small file takes less than a millisecond instead of 60 ms. - Fix
igzip_threadedwriters that waited forever when writing to the underlying file failed, for example, because the disk was full. A later call towrite(),flush(), orclose()raises the error. After an error,close()no longer closes a file object that you passed toigzip_threaded.open. - Fix
igzip_threadedwriters with more than one thread that waited forever when compressing a block failed, for example, because there was not enough memory. A later call towrite(),flush(), orclose()raises the error. - Fix a crash on the free-threaded build when one thread reads
unused_dataorunconsumed_tailof anisal_zlibdecompressor while another thread decompresses with it. Reading the attributes ofisal_zlibdecompressors andigzip_lib.IgzipDecompressorfrom other threads is now safe, and so is callingdecompress()from several threads at the same time whenwbitsdetects the format. isal_zlib.crc32_combineand theread()andseek()methods of the privateisal_zlib._GzipReaderclass raiseTypeErrorinstead ofSystemErrorfor arguments of the wrong type or number.igzip.IGzipFilewas not affected.- Fix a crash on PyPy when you create an
isal_zlib.Compressorisal_zlib.Decompressobject by calling the type, for example,isal_zlib.Compress(), and then use it. Calling the type raisesTypeErrornow, as it does on CPython. Usecompressobj()anddecompressobj()instead. - Fix
readinto()of the privateisal_zlib._GzipReaderclass, which wrote into read-only objects such asbytes. It raisesTypeErrornow.igzip.IGzipFilewas not affected. - Fix the
mtimeattribute ofigzip.IGzipFileobjects in read mode, which wasNonefor gzip files with a modification time of 0. It is 0 now, as in thegzipmodule.gzip -n,python -m isal.igzip -n, andigzip_threadedwriters make such files, and so doesgzip.compressby default on Python 3.14. - Fix a crash when there is not enough memory to create an
igzip_lib.IgzipDecompressor. It raisesMemoryErrornow. - Fix a crash when there is not enough memory to raise
BadGzipFilefor data that is not in the gzip format, for example, inigzip.decompress. It raisesMemoryErrornow. - Fix a crash in
igzip_lib.IgzipDecompressor.decompress()after a call on the same object raisedMemoryError. The next call used the input of the failed call after the caller had released it. This matches the CPython fix for gh-148395. - Fix a crash on macOS and PyPy when there is not enough memory to create the
internal lock of an
isal_zlib.Compress,isal_zlib.Decompress, origzip_lib.IgzipDecompressorobject. It raisesMemoryErrornow. - Fix the
-0and--fastoptions ofpython -m isal.igzip, which compressed at level 2 instead of level 0. - Fix crashes and wrong data when one thread calls
seek()of the privateisal_zlib._GzipReaderclass while another thread reads from the same object.igzip.IGzipFilewas not affected.
- Python 3.14 is supported.
- Python 3.8 and 3.9 are no longer supported.
- Fix an issue where flushing using igzip_threaded caused a gzip end of stream and started a new gzip stream. In essence creating a concatenated gzip stream. Now it is in concordance with how single threaded gzip streams are flushed using Z_SYNC_FLUSH.
- Change build backend to setuptools-scm which is more commonly used and supported.
- Include test packages in the source distribution, so source distribution installations can be verified.
- Fix an issue where some tests failed because they ignored PYTHONPATH.
- Enable support for free-threading and build free-threaded wheels for CPython 3.14. Thanks to @lysnikolaou and @ngoldbaum.
- Use upstream ISA-L version 2.31.1 which includes patches to make installation on MacOS ARM64 possible.
- Fix a bug where bytes were copied in the wrong order on big endian architectures. Fixes test failures on s390x.
- Enable building on GNU/Hurd platforms.
- Fix a bug where flushing files when writing in threaded mode did not work properly.
- Prevent threaded opening from blocking python exit when an error is thrown in the calling thread.
- Include a patched ISA-L version 2.31. The applied patches make compilation and wheelbuilding on MacOS ARM64 possible.
- Fix a bug where READ and WRITE in isal.igzip were inconsistent with the values in gzip on Python 3.13
- Small simplifications to the
igzip.compressfunction, which should lead to less overhead.
- Fix a bug where streams that were passed to igzip_threaded.open where closed.
- Fix a bug where compression levels for IGzipFile where checked in read mode.
- Update statically linked ISA-L release to 2.31.0
- Fix an error that occurred in the
__close__function when a threaded writer was initialized with incorrect parameters.
- Fix a bug where append mode would not work when using
igzip_threaded.open.
- Fix a bug where a filehandle remained opened when
igzip_threaded.openwas used for writing with a wrong compression level. - Fix a memory leak that occurred when an error was thrown for a gzip header with the wrong magic numbers.
- Fix a memory leak that occurred when isal_zlib.decompressobj was given a wrong wbits value.
- Fix a memory leak in the GzipReader.readall implementation.
- Make a special case for threads==1 in
igzip_threaded.openfor writing files. This now combines the writing and compression thread for less overhead. - Maximize time spent outside the GIL for
igzip_threaded.openwriting. This has decreased wallclock time significantly.
- Fix several errors related to unclosed files and buffers.
- Drop support for python 3.7 and PyPy 3.8 as these are no longer supported. Add testing and support for python 3.12 and PyPy 3.10.
- Added an experimental
isal.igzip_threadedmodule which has anopenfunction. This can be used to read and write large files in a streaming fashion while escaping the GIL. - The internal
igzip._IGzipReaderhas been rewritten in C. As a result the overhead of decompressing files has significantly been reduced andpython -m isal.igzipis now very close to the Cigzipapplication. - The
igzip._IGZipReaderin C is now used inigzip.decompress. The_GzipReaderalso can read from objects that support the buffer protocol. This has reduced overhead significantly.
- Gzip headers are now actively checked for a BGZF extra field. If found the block size is taken into account when decompressing. This has further improved bgzf decompression speed by 5% on some files compared to the more generic solution of 1.2.0.
- Integrated CPython 3.11 code for reading gzip headers. This leads to more
commonality between the python-isal code and the upstream gzip.py code.
This has enabled the change above. It comes at the cost of a slight increase
in overhead at the
gzip.decompressfunction.
- Bgzip files are now detected and a smaller reading buffer is used to accomodate the fact that bgzip blocks are typically less than 64K. (Unlike normal gzip files that consist of one block that spans the entire file.) This has reduced decompression time for bgzip files by roughly 12%.
- Speed-up source build by using ISA-L Unix-specific makefile rather than the autotools build.
- Simplify build setup. ISA-L release flags are now used and not overwritten with python release flags when building the included static library.
- Fix bug where zdict's could not be set for
isal_zlib.decompressobjandigzip_lib.IgzipDecompressor. - Escape GIL when calling inflate, deflate, crc32 and adler32 functions just like in CPython. This allows for utilising more CPU cores in combination with the threading module. This comes with a very slight cost in efficiency for strict single-threaded applications.
- Added tests and support for Python 3.11.
- Fixed failing tests and wheel builds for PyPy.
Python-isal has been rewritten as a C-extension (first implementation was in Cython). This has made the library faster in many key areas.
- Since the module now mostly contains code copied from CPython and then modified to work with ISA-L the license has been changed to the Python Software Foundation License version 2.
- Python versions lower than 3.7 are no longer supported. Python 3.6 is out of support since December 2021.
- Stub files with type information have now been updated to correctly display positional-only arguments.
- Expose
READandWRITEconstants on theigzipmodule. These are also present in Python's stdlibgzipmodule and exposing them allows for better drop-in capability ofigzip. Thanks to @alexander-beedie in #115. - A
--no-nameflag has been added topython -m isal.igzip. - Reduced wheel size by not including debug symbols in the binary. Thanks to @marcelm in #108.
- Cython is no longer required as a build dependency.
- isal_zlib.compressobj and isal_zlib.decompressobj are now about six times faster.
- igzip.decompress has 30% less overhead when called.
- Error structure has been simplified. There is only
IsalErrorwhich hasExceptionas baseclass instead ofOSError.isal_zlib.IsalError,igzip_lib.IsalError,isal_zlib.errorandigzip_lib.errorare all aliases of the same error class. - GzipReader now uses larger input and output buffers (128k) by default and
IgzipDecompressor.decompress has been updated to allocate
maxsizebuffers when these are of reasonable size, instead of growing the buffer to maxsize on every call. This has improved gzip decompression speeds by 7%. - Patch statically linked included library (ISA-L 2.30.0) to fix the following:
- ISA-L library version variables are now available on windows as well, for the statically linked version available on PyPI.
- Wheels are now always build with nasm for the x86 architecture. Previously yasm was used for Linux and MacOS due to build issues.
- Fixed a bug upstream in ISA-L were zlib headers would be created with an incorrect wbits value.
- Python-isal shows up in Python profiler reports.
- Support and tests for Python 3.10 were added.
- Due to a change in the deployment process wheels should work for older versions of pip.
- Added a
crcproperty to the IgzipDecompressor class. Depending on the decompression flag chosen, this will update with an adler32 or crc32 checksum. - All the decompression NO_HDR flags on igzip_lib were incorrectly documented. This is now fixed.
- Fixed an issue which occurred rarely that caused IgzipDecompressor's unused_data to report back incorrectly. This caused checksum errors when reading gzip files. The issue was more likely to trigger in multi-member gzip files.
In this release the python -m isal.igzip relatively slow decompression rate
has been improved in both speed and usability. Previously it was 19% slower
than igzip when used with the -d flag for decompressing, now it is
just 8% slower. Also some extra flags were added to make it easier to select
the output file.
- Prompt when an output file is overwritten with the
python -m isal.igzipcommand line utility and provide the-for--forceflags to force overwriting. - Added
-oand--outputflags to thepython -m isal.igzipcommand line utility to allow the user to select the destination of the output file. - Reverse a bug in the build system which caused some docstring and parameter
information on
igzip_libandisal_zlibto disappear in the documentation and the REPL. - Increase the buffer size for
python -m isal.igzipso it is now closer to speeds reached withigzip. - Add a
READ_BUFFER_SIZEattribute toigzipwhich allows setting the amount of raw data that is read at once. - Add an
igzip_lib.IgzipDecompressorobject which can decompress without using an unconsumed_tail and is therefore more efficient.
- Added an
igzip_libmodule which allows more direct access to ISA-L's igzip_lib API. This allows features such as headerless compression and decompression, as well as setting the memory levels manually. - Added more extensive documentation.
- Fix a bug where a AttributeError was triggered when zlib.Z_RLE or zlib.Z_FIXED were not present.
- Add support for Linux aarch64 builds.
- Add support for pypy by adding pypy tests to the CI and setting up wheel building support.
- Fix a bug where multi-member gzip files where read incorrectly due to an offset error. This was caused by ISA-L's decompressobj having a small bitbuffer which was not taken properly into account in some circumstances.
- Speed up
igzip.compressandigzip.decompressby improving the implementation. - Make sure compiler arguments are passed to ISA-L compilation step. Previously ISA-L was compiled without optimisation steps, causing the statically linked library to be significantly slower.
- A unused constant from the
isal_zliblibrary was removed:ISAL_DEFAULT_HIST_BITS. - Refactor isal_zlib.pyx to work almost the same as zlibmodule.c. This has made the code look cleaner and has reduced some overhead.
- Remove workarounds in the
igzipmodule for theunconsumed_tailandunused_databugs.igzip._IGzipReadernow functions the same asgzip._GzipReaderwith only a few calls replaced withisal_zlibcalls for speed. - Correctly implement
unused_dataandunconsumed_tailonisal_zlib.Decompressobjects. It works the same as in CPython's zlib now. - Correctly implement flush implementation on
isal_zlib.Compressandisal_zlib.Decompressobjects. It works the same as in CPython's zlib now.
- Fix a crash that occurs when opening a file that did not end in
.gzwhile outputting to stdout usingpython -m isal.igzip.
python -m gzip's behaviour has been changed since fixing bug: bpo-43316. This bug was not present inpython -m isal.igzipbut it handled the error differently than the solution in CPython. This is now corrected andpython -m isal.igziphandles the error the same as the fixedpython -m gzip.- Installation on Windows is now supported. Wheels are provided for Windows as well.
- Fix a bug where negative integers were not allowed for the
adler32andcrc32functions inisal_zlib. - Provided stubs (type-hint files) for
isal_zliband_isalmodules. Package is now tested with mypy to ensure correct type information. - The command-line interface now reads in blocks of 32K instead of 8K. This
improves performance by about 6% when compressing and 11% when decompressing.
A hidden
-bflag was added to adjust the buffer size for benchmarks. - A
-cor--stdoutflag was added to the CLI interface of isal.igzip. This allows it to behave more like thegziporpigzcommand line interfaces.
- Move wheel building to cibuildwheel on github actions CI. Wheels are now provided for Mac OS as well.
- Make a tiny change in setup.py so python-isal can be build on Mac OS X.
- Set included ISA-L library at version 2.30.0.
- Python-isal now comes with a source distribution of ISA-L in its source distribution against which python-isal is linked statically upon installation by default. Dynamic linking against system libraries is now optional. Wheels with the statically linked ISA-L are now provided on PyPI.
- Fixed a bug where writing of the gzip header would crash if an older version of Python 3.7 was used such as on Debian or Ubuntu. This is due to differences between point releases because of a backported feature. The code now checks if the backported feature is present.
- Added Python 3.9 to the testing.
- Fixed
setup.pyto list setuptools as a requirement. - Changed homepage to reflect move to pycompression organization.
- Publish API documentation on readthedocs.
- Add API documentation.
- Ensure the igzip module is fully compatible with the gzip stdlib module.
- Add compliance tests from CPython to ensure isal_zlib and igzip are validated to the same standards as the zlib and gzip modules.
- Added a working gzip app using
python -m isal.igzip - Add test suite that tests all possible settings for functions on the isal_zlib module.
- Create igzip module which implements all gzip functions and methods.
- Create isal_zlib module which implements all zlib functions and methods.