Repository navigation
Return b"" from Compress.flush() for a repeated sync or full flush, as zlib does - #330
Conversation
A sync or full flush with nothing compressed since the previous flush made ISA-L write an empty block: 6 bytes at levels 1 to 3, and 117 bytes at level 0, where each block starts with a Huffman table. Each such flush also allocated a 16 KiB output buffer and called isal_deflate. zlib returns b"" in this case. Compress objects now remember the mode of their last flush. A sync flush after a sync or full flush, and a full flush after a full flush, return b"" when no compress() call came in between. A full flush after a sync flush still writes a block, because it resets the history. compress() forgets the mode even for empty data, as deflate() does in zlib, so compress(b"") + flush(Z_SYNC_FLUSH) still ends in 00 00 ff ff, which permessage-deflate needs for empty messages. Z_FINISH forgets it too, so flushes after finishing behave as before. pyperf timeit on Linux, Python 3.11 / 3.14, mean of 2 runs: - repeated flush(Z_SYNC_FLUSH), level 2: 1.49 / 1.38 µs -> 47 / 51 ns - repeated flush(Z_SYNC_FLUSH), level 0: 160 / 162 ns -> 46 / 50 ns - repeated IGzipFile.flush(): 1.63 / 1.66 µs -> 229 / 323 ns One write followed by 100 IGzipFile.flush() calls now gives 29 instead of 623 bytes at levels 1 to 3 (gzip gives 30), and 253 instead of 11836 bytes at level 0. A flush after new data costs 8 more instructions, which is within timing noise. The output changes only for back-to-back flushes, and the stream stays valid.
8457fec to
1e8b2a4
Compare
|
Interesting. How is this handled in CPython? Is this just a difference between the zlib and isa-l flush implementations or is this a difference between python-isal and CPython's zlibmodule.c? |
zlib's deflate() stores the flush mode of every call, including the calls it skips, so after a full flush and a skipped sync flush, another full flush writes a block. Compress.flush() kept the mode of the full flush instead and returned b"" for the second full flush. It now also stores the mode of a skipped flush, so it returns b"" for exactly the flushes for which zlib writes nothing. A new test compares isal_zlib with zlib for every sequence of up to four compress() and flush() calls, at every level and for raw, zlib and gzip streams.
|
It's a difference between the zlib and ISA-L libraries. CPython's zlibmodule.c does nothing special here. CPython's
I just pushed aeff64c, which also records the mode of a skipped flush, as Calling both libraries directly from C, with no Python involved, on raw deflate after compressing 35 bytes and one sync flush:
ISA-L returns Test program#include <stdio.h>
#include <stdlib.h>
#include <zlib.h>
#include <igzip_lib.h>
static unsigned char data[] = "hello hello hello hello hello hello";
static unsigned char out[65536];
static const char *names[] = {"data", "sync", "sync again", "full",
"full again", "sync"};
int main(void)
{
int zlib_flush[] = {Z_NO_FLUSH, Z_SYNC_FLUSH, Z_SYNC_FLUSH,
Z_FULL_FLUSH, Z_FULL_FLUSH, Z_SYNC_FLUSH};
int isal_flush[] = {NO_FLUSH, SYNC_FLUSH, SYNC_FLUSH,
FULL_FLUSH, FULL_FLUSH, SYNC_FLUSH};
uint32_t level_buf_size[] = {ISAL_DEF_LVL0_DEFAULT, ISAL_DEF_LVL1_DEFAULT,
ISAL_DEF_LVL2_DEFAULT, ISAL_DEF_LVL3_DEFAULT};
z_stream z = {0};
int i, rc, level;
deflateInit2(&z, 6, Z_DEFLATED, -15, 8, Z_DEFAULT_STRATEGY);
printf("zlib %s\n", zlibVersion());
for (i = 0; i < 6; i++) {
z.next_in = data;
z.avail_in = i == 0 ? sizeof(data) - 1 : 0;
z.next_out = out;
z.avail_out = sizeof(out);
rc = deflate(&z, zlib_flush[i]);
printf(" %-10s rc=%2d %3u bytes\n", names[i], rc,
(unsigned)(sizeof(out) - z.avail_out));
}
deflateEnd(&z);
for (level = 0; level <= 3; level++) {
struct isal_zstream s;
isal_deflate_init(&s);
s.level = level;
s.level_buf_size = level_buf_size[level];
s.level_buf = malloc(level_buf_size[level] ? level_buf_size[level] : 1);
printf("ISA-L level %d\n", level);
for (i = 0; i < 6; i++) {
s.next_in = data;
s.avail_in = i == 0 ? sizeof(data) - 1 : 0;
s.flush = isal_flush[i];
s.next_out = out;
s.avail_out = sizeof(out);
rc = isal_deflate(&s);
printf(" %-10s rc=%2d %3u bytes\n", names[i], rc,
(unsigned)(sizeof(out) - s.avail_out));
}
free(s.level_buf);
}
return 0;
}
|
The comparison with zlib runs every sequence of up to four compress() and flush() calls at every level, for raw, zlib and gzip streams. That covers what the separate tests checked: repeated sync and full flushes, a flush before any data, a full flush after a sync flush and a flush after compress(b""). Remove those tests, and the repeated sync case in test_dictionary_header.py, since a dictionary does not change which flushes are skipped.
|
Thanks for all the explanation. This is a great compatibility fix. Thanks a lot! |
A
Z_SYNC_FLUSHorZ_FULL_FLUSHwith nothing compressed since the previous flush makes ISA-L write an empty block: 6 bytes at levels 1 to 3, and 117 bytes at level 0, where every block starts with a Huffman table. Each such flush also allocates a 16 KiB output buffer. zlib writes nothing in this case:deflate()returnsZ_BUF_ERROR, and CPython'sCompress.flush()returnsb"". ISA-L documents the difference inigzip_lib.hsince v2.32.1 and suggests that callers skip such flushes.IGzipFile.flush()callsflush(Z_SYNC_FLUSH), so a program that flushes a gzip file often without writing in between makes the file grow with every flush. Ordinary text-mode use triggers it too:TextIOWrapper.close()flushes again, so a file fromigzip.open(..., "wt")that is flushed and then closed, for example byprint(..., flush=True)or alogging.StreamHandler, gets an empty block thatgzipdoes not write.With this PR,
Compressobjects remember the mode of their last flush, asdeflate()does:b"". As in zlib, a skipped flush counts as well, so a full flush that follows a full and a skipped sync flush writes a block.compress()forgets the mode even forb"", as zlib'sdeflate()does, socompress(b"")followed byflush(Z_SYNC_FLUSH)still ends in00 00 ff ff. permessage-deflate needs that for empty messages.Z_FINISHforgets it too, so flushes after finishing work as before.These are the rules of zlib's
deflate(), soisal_zlibnow returnsb""for exactly the flushes for which zlib writes nothing. The C change adds a field and an early return toCompress.flush(), and does not touch the output buffer code that the move to the stable ABI (#240) will replace.I measured with pyperf in Docker on Linux, as the mean of 2 runs on Python 3.11 and 3.14:
develop(3.11 / 3.14)flush(Z_SYNC_FLUSH), level 2flush(Z_SYNC_FLUSH), level 0IGzipFile.flush()One write followed by 100
IGzipFile.flush()calls now gives 29 bytes instead of 623 at levels 1 to 3 (thegzipmodule gives 30), and 253 bytes instead of 11836 at level 0. A flush after new data costs 8 more instructions out of about 55,000 for acompress()of 100 bytes and aflush(), which wall-clock time does not show.The output changes only for back-to-back flushes, and the stream stays valid. One new test checks that
isal_zlibreturnsb""for the same flushes as zlib, for every sequence of up to fourcompress()andflush()calls at every level, for raw, zlib and gzip streams, and that the result decompresses. Another flushes anIGzipFiletwice in a row. Both fail ondevelop. The test suite passes on this branch on Linux with CPython 3.14, and CI covers the other versions.Checklist