Repository navigation
perf(cpp): batch and SIMD decode for the RLE bit-packing codecs - #1078
Merged
Merged
Conversation
RLE was the last hot value codec without a batch path: Plain, Gorilla and TS_2DIFF got read_batch_* in apache#823/apache#898, while RLE still decoded value by value through read_int and unpacked each bit-packing group with the portable scalar packer. On a wide table the RLE decoder dominates the read profile. - add read_batch_int32/int64 and read_batch_float/double: an RLE run becomes a std::fill, a bit-packing group becomes one memcpy out of the unpacked buffer. - unpack four values per step with a SIMD gather+shuffle kernel and dispatch to native AVX2 intrinsics at runtime (__builtin_cpu_supports) instead of relying on the translation unit being compiled with -mavx2; the simde fallback stays for other targets, and the widths that can cross a ninth byte (59/61/62/63) use a 128-bit buffer where the compiler has one. - read each bit-packing group with one bulk read instead of per-byte read_ui8, and wrap the caller's stream for the pattern length instead of copying it. - the SIMD gather reads up to 7 bytes past the last group, so the RLE and Sprintz decoders pad their buffers by 8 zero bytes; the packer tests do the same and pass the exact packed length. - guard the 128-bit scalar path with __SIZEOF_INT128__ and keep the portable bit loop on MSVC.
A corrupt bit-packing header could size the new bulk read with a wrapped uint32 product, turn bytes_to_read negative, and write the rest of the pattern past a 7-byte scratch vector. Validate the bit width, group byte count and last_bit_packed_num before sizing anything, and reject patterns whose declared groups do not fit instead of clamping them. - read_bit_packing_buffer: compute group_bytes in 64-bit, bound it by the remaining pattern bytes and mem_alloc's uint32 argument; validate bit_width_ (1..32 / 1..64). - call_read_bit_packing_buffer: require last_bit_packed_num in [1,8], bound the value count by INT32_MAX, and clear the count when the group load fails; read_batch_* now refuses a non-positive count instead of passing it to memcpy. - read_length_and_bitwidth: reject oversized length prefixes, check the bit-width read, and drop the borrowed byte_cache_ view on reset()/destroy(); the int64 decoder deletes the old packer per pattern. - unpack_all_values now decodes the final group from a zero-padded copy so callers no longer need the +8 tail; guard zero-width non-advancing loops. - hoist the simde shuffle mask to a constant byte array, removing the thread-safe-static guard from every kernel call. - Sprintz decoders reject out-of-range block widths. Add differential batch/scalar tests (wrapped and owning streams, capacities 1..1024, mixed run and bit-packed segments, cross-pattern batches, reset and reuse), malformed-header rejection tests, Float/Double RLE batch tests, and exact-size packer tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
RLE was the last hot value codec without a batch path:
Plain,GorillaandTS_2DIFFreceivedread_batch_*in #823/#898, while RLE still decoded valueby value through
read_intand unpacked each bit-packing group with theportable scalar packer. On a wide table (thousands of measurement columns per
device) the RLE decoder dominates the read profile.
read_batch_int32/read_batch_int64andread_batch_float/read_batch_double. An RLE run becomes astd::fill, abit-packing group becomes one
memcpyout of the unpacked buffer.a native AVX2 path dispatched at runtime via
__builtin_cpu_supportsinsteadof relying on the translation unit being compiled with
-mavx2. The portablesimde fallback stays for other targets. Widths that can cross a ninth byte
(59/61/62/63) use a 128-bit buffer where the compiler supports
__int128.of per-byte
read_ui8, and the pattern length wraps the caller'sByteStreaminstead of copying it into a temporary buffer.
Correctness notes
must hold 8 readable bytes of padding after the final group. The RLE decoders
already zero-pad; the Sprintz block decoders now do too, and the packer tests
pad their buffers and pass the exact packed length.
__int128, so the 128-bit scalar path is guarded by__SIZEOF_INT128__with the portable bit-by-bit loop as the fallback.Measurements
Unpack micro-benchmark, values/s,
-O2without-mavx2(runtime dispatchtakes the AVX2 path on the test machine):
End-to-end on a wide-table workload (5 features x 4500 measurement columns x
1 month of minute data): this step alone took a single query from 3.14 s to
2.38 s.
Testing
-Denable.asan=ON(Debug) aspart of a larger branch, including LeakSanitizer.