Skip to content

Return b"" from Compress.flush() for a repeated sync or full flush, as zlib does - #330

Merged
rhpvorderman merged 6 commits into
pycompression:developfrom
ptweezy:redundant-sync-flush
Nov 18, 2026
Merged

rhpvorderman merged 6 commits into
pycompression:developfrom
ptweezy:redundant-sync-flush

Conversation

@ptweezy

@ptweezy ptweezy commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

A Z_SYNC_FLUSH or Z_FULL_FLUSH with nothing compressed since the previous flush makes ISA-L write an empty block: 6 bytes at levels 1 to 3, and 117 bytes at level 0, where every block starts with a Huffman table. Each such flush also allocates a 16 KiB output buffer. zlib writes nothing in this case: deflate() returns Z_BUF_ERROR, and CPython's Compress.flush() returns b"". ISA-L documents the difference in igzip_lib.h since v2.32.1 and suggests that callers skip such flushes. IGzipFile.flush() calls flush(Z_SYNC_FLUSH), so a program that flushes a gzip file often without writing in between makes the file grow with every flush. Ordinary text-mode use triggers it too: TextIOWrapper.close() flushes again, so a file from igzip.open(..., "wt") that is flushed and then closed, for example by print(..., flush=True) or a logging.StreamHandler, gets an empty block that gzip does not write.

With this PR, Compress objects remember the mode of their last flush, as deflate() does:

  • A sync flush right after a sync or full flush, or a full flush right after a full flush, returns b"". As in zlib, a skipped flush counts as well, so a full flush that follows a full and a skipped sync flush writes a block.
  • A full flush after a sync flush still writes a block, because it resets the history.
  • compress() forgets the mode even for b"", as zlib's deflate() does, so compress(b"") followed by flush(Z_SYNC_FLUSH) still ends in 00 00 ff ff. permessage-deflate needs that for empty messages.
  • Z_FINISH forgets it too, so flushes after finishing work as before.

These are the rules of zlib's deflate(), so isal_zlib now returns b"" for exactly the flushes for which zlib writes nothing. The C change adds a field and an early return to Compress.flush(), and does not touch the output buffer code that the move to the stable ABI (#240) will replace.

I measured with pyperf in Docker on Linux, as the mean of 2 runs on Python 3.11 and 3.14:

Scenario develop (3.11 / 3.14) This PR (3.11 / 3.14)
Repeated flush(Z_SYNC_FLUSH), level 2 1.49 / 1.38 µs 47 / 51 ns
Repeated flush(Z_SYNC_FLUSH), level 0 160 / 162 ns 46 / 50 ns
Repeated IGzipFile.flush() 1.63 / 1.66 µs 229 / 323 ns

One write followed by 100 IGzipFile.flush() calls now gives 29 bytes instead of 623 at levels 1 to 3 (the gzip module gives 30), and 253 bytes instead of 11836 at level 0. A flush after new data costs 8 more instructions out of about 55,000 for a compress() of 100 bytes and a flush(), which wall-clock time does not show.

The output changes only for back-to-back flushes, and the stream stays valid. One new test checks that isal_zlib returns b"" for the same flushes as zlib, for every sequence of up to four compress() and flush() calls at every level, for raw, zlib and gzip streams, and that the result decompresses. Another flushes an IGzipFile twice in a row. Both fail on develop. The test suite passes on this branch on Linux with CPython 3.14, and CI covers the other versions.

Checklist

  • Pull request details were added to CHANGELOG.rst
  • Documentation was updated (if needed)

A sync or full flush with nothing compressed since the previous flush
made ISA-L write an empty block: 6 bytes at levels 1 to 3, and 117
bytes at level 0, where each block starts with a Huffman table. Each
such flush also allocated a 16 KiB output buffer and called
isal_deflate. zlib returns b"" in this case.

Compress objects now remember the mode of their last flush. A sync
flush after a sync or full flush, and a full flush after a full flush,
return b"" when no compress() call came in between. A full flush after
a sync flush still writes a block, because it resets the history.
compress() forgets the mode even for empty data, as deflate() does in
zlib, so compress(b"") + flush(Z_SYNC_FLUSH) still ends in 00 00 ff ff,
which permessage-deflate needs for empty messages. Z_FINISH forgets it
too, so flushes after finishing behave as before.

pyperf timeit on Linux, Python 3.11 / 3.14, mean of 2 runs:
- repeated flush(Z_SYNC_FLUSH), level 2: 1.49 / 1.38 µs -> 47 / 51 ns
- repeated flush(Z_SYNC_FLUSH), level 0: 160 / 162 ns -> 46 / 50 ns
- repeated IGzipFile.flush(): 1.63 / 1.66 µs -> 229 / 323 ns
One write followed by 100 IGzipFile.flush() calls now gives 29 instead
of 623 bytes at levels 1 to 3 (gzip gives 30), and 253 instead of 11836
bytes at level 0. A flush after new data costs 8 more instructions,
which is within timing noise.

The output changes only for back-to-back flushes, and the stream stays
valid.
@ptweezy
ptweezy force-pushed the redundant-sync-flush branch from 8457fec to 1e8b2a4 Compare November 6, 2026 23:27
@rhpvorderman

Copy link
Copy Markdown
Collaborator

Interesting. How is this handled in CPython? Is this just a difference between the zlib and isa-l flush implementations or is this a difference between python-isal and CPython's zlibmodule.c?

zlib's deflate() stores the flush mode of every call, including the
calls it skips, so after a full flush and a skipped sync flush, another
full flush writes a block. Compress.flush() kept the mode of the full
flush instead and returned b"" for the second full flush. It now also
stores the mode of a skipped flush, so it returns b"" for exactly the
flushes for which zlib writes nothing.

A new test compares isal_zlib with zlib for every sequence of up to
four compress() and flush() calls, at every level and for raw, zlib and
gzip streams.
@ptweezy

ptweezy commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

It's a difference between the zlib and ISA-L libraries. CPython's zlibmodule.c does nothing special here.

CPython's Compress.flush() sets avail_in = 0, calls deflate(&self->zst, mode) and treats Z_BUF_ERROR as success (zlibmodule.c). The b"" comes from deflate() itself. It keeps the flush mode of the previous call in s->last_flush, and when there is no new input and the flush is not stronger than that one, it returns Z_BUF_ERROR without writing anything (deflate.c). zlib has done this since version 0.99 in 1996. zlib-ng has the same check (deflate.c), so CPython 3.14 on Windows and python-zlib-ng return b"" the same way.

isal_deflate() keeps no such state. ISA-L documents this as a difference from zlib in a note added in v2.32.1, the version in the submodule (igzip_lib.h). A sync or full flush always writes the sync marker, even without new input, and callers that flush twice should expect the extra bytes or "skip the call when no input is pending". python-isal's flush() passes the mode to isal_deflate() the way CPython passes it to deflate(), so it gets ISA-L's behavior. This PR skips the call where zlib writes nothing.

I just pushed aeff64c, which also records the mode of a skipped flush, as deflate() does. That removes the one exception the description used to mention, and a new test checks this against zlib for every sequence of up to four compress() and flush() calls. I also merged develop to fix the changelog conflict.

Calling both libraries directly from C, with no Python involved, on raw deflate after compressing 35 bytes and one sync flush:

Next calls, in order zlib 1.3.1 ISA-L 2.32.1, levels 1 to 3 ISA-L 2.32.1, level 0
sync flush 0 bytes (Z_BUF_ERROR) 6 bytes 117 bytes
full flush 5 bytes 6 bytes 117 bytes
full flush 0 bytes (Z_BUF_ERROR) 6 bytes 117 bytes
sync flush 0 bytes (Z_BUF_ERROR) 6 bytes 117 bytes

ISA-L returns COMP_OK every time. The 6 bytes are 02 00 00 00 ff ff, an empty fixed Huffman block followed by the empty stored block. At level 0, every block starts with a Huffman table.

Test program
#include <stdio.h>
#include <stdlib.h>
#include <zlib.h>
#include <igzip_lib.h>

static unsigned char data[] = "hello hello hello hello hello hello";
static unsigned char out[65536];
static const char *names[] = {"data", "sync", "sync again", "full",
                              "full again", "sync"};

int main(void)
{
    int zlib_flush[] = {Z_NO_FLUSH, Z_SYNC_FLUSH, Z_SYNC_FLUSH,
                        Z_FULL_FLUSH, Z_FULL_FLUSH, Z_SYNC_FLUSH};
    int isal_flush[] = {NO_FLUSH, SYNC_FLUSH, SYNC_FLUSH,
                        FULL_FLUSH, FULL_FLUSH, SYNC_FLUSH};
    uint32_t level_buf_size[] = {ISAL_DEF_LVL0_DEFAULT, ISAL_DEF_LVL1_DEFAULT,
                                 ISAL_DEF_LVL2_DEFAULT, ISAL_DEF_LVL3_DEFAULT};
    z_stream z = {0};
    int i, rc, level;

    deflateInit2(&z, 6, Z_DEFLATED, -15, 8, Z_DEFAULT_STRATEGY);
    printf("zlib %s\n", zlibVersion());
    for (i = 0; i < 6; i++) {
        z.next_in = data;
        z.avail_in = i == 0 ? sizeof(data) - 1 : 0;
        z.next_out = out;
        z.avail_out = sizeof(out);
        rc = deflate(&z, zlib_flush[i]);
        printf("  %-10s rc=%2d %3u bytes\n", names[i], rc,
               (unsigned)(sizeof(out) - z.avail_out));
    }
    deflateEnd(&z);

    for (level = 0; level <= 3; level++) {
        struct isal_zstream s;
        isal_deflate_init(&s);
        s.level = level;
        s.level_buf_size = level_buf_size[level];
        s.level_buf = malloc(level_buf_size[level] ? level_buf_size[level] : 1);
        printf("ISA-L level %d\n", level);
        for (i = 0; i < 6; i++) {
            s.next_in = data;
            s.avail_in = i == 0 ? sizeof(data) - 1 : 0;
            s.flush = isal_flush[i];
            s.next_out = out;
            s.avail_out = sizeof(out);
            rc = isal_deflate(&s);
            printf("  %-10s rc=%2d %3u bytes\n", names[i], rc,
                   (unsigned)(sizeof(out) - s.avail_out));
        }
        free(s.level_buf);
    }
    return 0;
}

gcc repeated_flush.c -I isa-l/include isa-l/bin/isa-l.a -lz, with ISA-L 2.32.1 from the submodule. I used the portable C build (arch=noarch).

The comparison with zlib runs every sequence of up to four compress()
and flush() calls at every level, for raw, zlib and gzip streams. That
covers what the separate tests checked: repeated sync and full
flushes, a flush before any data, a full flush after a sync flush and
a flush after compress(b""). Remove those tests, and the repeated sync
case in test_dictionary_header.py, since a dictionary does not change
which flushes are skipped.
@rhpvorderman

Copy link
Copy Markdown
Collaborator

Thanks for all the explanation. This is a great compatibility fix. Thanks a lot!

@rhpvorderman
rhpvorderman merged commit 3328341 into pycompression:develop Nov 18, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants