Repository navigation
Conversation
xopen reads gzip files with python-isal whenever it is installed. ISA-L decompresses faster than zlib-ng on x86-64, but on aarch64 zlib-ng is faster. Prefer python-isal for reading on x86-64 only, so that other architectures use python-zlib-ng when it is installed. python-isal is still used on all architectures for writing at compression levels 1 and 2.
The isal dependency marker lists x86_64, AMD64 and aarch64, but macOS on Apple Silicon reports arm64, so python-isal was never installed there. python-isal 1.9.0 builds ISA-L's aarch64 assembly on that platform instead of the portable C code, so require that version for arm64. Reading is unchanged on that platform because python-isal is only used for reading on x86-64.
rhpvorderman
left a comment
There was a problem hiding this comment.
I do not like the pattern of creating an unnecessary module level variable, just so that it can be monkeypatched.
Better to make the _use_isal function opaque and monkeypatch the attributes it tests for that relate to system architecture.
| if sys.platform == "win32": | ||
| _MACHINE = os.environ.get("PROCESSOR_ARCHITECTURE", "AMD64") |
There was a problem hiding this comment.
Getting this info from the environment seems awfully flaky... Isn't there some winapi call for this? Or some python function?
| # ISA-L decompresses faster than zlib-ng and zlib on x86-64. On ARM (aarch64 on | ||
| # Linux, arm64 on macOS), zlib-ng decompresses faster than ISA-L, and so does | ||
| # the zlib that Apple ships with macOS. platform.machine() is not used here | ||
| # because it takes tens of milliseconds on Windows. | ||
| if sys.platform == "win32": | ||
| _MACHINE = os.environ.get("PROCESSOR_ARCHITECTURE", "AMD64") | ||
| else: | ||
| _MACHINE = os.uname().machine | ||
| _ISAL_DECOMPRESSES_FASTEST = _MACHINE.lower() in ("x86_64", "amd64") |
There was a problem hiding this comment.
This code can just be in _use_isal. It makes more sense there.
|
|
||
| if xopen.igzip is None or xopen.gzip_ng is None: | ||
| pytest.skip("requires python-isal and python-zlib-ng") | ||
| monkeypatch.setattr(xopen, "_ISAL_DECOMPRESSES_FASTEST", isal_decompresses_fastest) |
There was a problem hiding this comment.
Rather than monkey patching global variables from the code it is better to monkeypatch the things that are checked in the _use_isal function, such as the sys.platorm variables.
xopen reads gzip files with python-isal whenever it is installed. That is the fastest choice on x86-64, but not on ARM. This PR makes xopen prefer python-isal for reading on x86-64 only, and installs python-isal on macOS with Apple Silicon, where it is now only used for writing.
Why
Reading and writing 222 MB of synthetic FASTQ (63 MB as
gzip -6) with each backend directly, in 128 KiB chunks, best of 7. Numbers are MB/s of uncompressed data, with isal 1.9.1 and zlib-ng 1.1.0 on Python 3.14, on GitHub-hosted runners:gzipisal.igzipzlib_ng.gzip_nggzip, level 1isal.igzip, level 1zlib_ng.gzip_ng, level 2So ISA-L is the fastest compressor everywhere, but it is the fastest decompressor on x86-64 only. On Linux aarch64, zlib-ng decompresses 1.6 times as fast as ISA-L. On macOS arm64, the
gzipmodule is the fastest reader, because that Python links to the zlib that Apple ships.Changes
pigz,gzipor thegzipmodule otherwise, as it already does without python-isal. python-isal is still used on all architectures for writing at levels 1 and 2. The architecture comes fromos.uname(), or fromPROCESSOR_ARCHITECTUREon Windows, becauseplatform.machine()took 34 ms on Windows when I measured it.isal>=1.9.0; platform.machine == "arm64". python-isal builds the aarch64 assembly of ISA-L on that platform since 1.9.0 (Build ISA-L's aarch64 assembly on macOS arm64 python-isal#268). With the first commit, this only changes writing there. On its own it would make reading slower, as the first table shows.Results
xopenonmainand on this branch, median of 7 alternating runs in new processes, in MB/s of uncompressed data. The real file isSRR2589044_1.fastq.gzfrom ENA (E. coli, Illumina, 407 MB of FASTQ), read as published.Linux aarch64 (Neoverse-N2), where both python-isal and python-zlib-ng are installed:
threadsmainReading in chunks is 1.5 to 1.6 times as fast. Iterating over lines in Python doesn't change in wall time. With the default
threads, its total CPU time drops by 12 to 16%.macOS arm64 (Apple M1), writing at level 1:
pigzinstalledthreadsmainmainWith
pigzinstalled and the defaultthreads, the wall time is about the same and the CPU time is less than half. The output is 3% smaller (71.9 instead of 74.0 MB, and 147.2 instead of 151.3 MB). Reading on macOS arm64 takes the same code path as onmain; the 16 read cases measured between 0.90x and 1.06x on the virtualized runner.On Linux x86-64 (Xeon 8573C), all 12 cases are between 0.99x and 1.02x.
Notes
pigz,gzipor thegzipmodule instead of python-isal. On the Neoverse-N2 runner, that isgzipat 450 instead ofisal.igzipat 514 MB/s. xopen installs both libraries on aarch64, so this needs python-zlib-ng to be removed by hand.no-libsenvironment, so CI doesn't test python-isal on macOS arm64. I rantox run -e 3.15andtox run -e no-libsfor this branch onmacos-latest(432 passed, 5 skipped and 428 passed, 9 skipped) and onubuntu-24.04-arm(437 passed and 429 passed, 8 skipped).The benchmark scripts are on the
bench-isal-macos-arm64branch of my fork. The numbers come from these runs: backends, xopen on Linux aarch64, macOS arm64 and Linux x86-64, and the tests on ARM.