Skip to content

perf(cpp): batch and SIMD decode for the RLE bit-packing codecs - #1078

Merged
ColinLeeo merged 2 commits into
apache:developfrom
ColinLeeo:perf/rle-simd
Nov 9, 2026
Merged

ColinLeeo merged 2 commits into
apache:developfrom
ColinLeeo:perf/rle-simd

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

What changes

RLE was the last hot value codec without a batch path: Plain, Gorilla and
TS_2DIFF received read_batch_* in #823/#898, while RLE still decoded value
by value through read_int and unpacked each bit-packing group with the
portable scalar packer. On a wide table (thousands of measurement columns per
device) the RLE decoder dominates the read profile.

  • Batch decode — read_batch_int32 / read_batch_int64 and
    read_batch_float / read_batch_double. An RLE run becomes a std::fill, a
    bit-packing group becomes one memcpy out of the unpacked buffer.
  • SIMD bit-unpack — four values per step with a gather+shuffle kernel, plus
    a native AVX2 path dispatched at runtime via __builtin_cpu_supports instead
    of relying on the translation unit being compiled with -mavx2. The portable
    simde fallback stays for other targets. Widths that can cross a ninth byte
    (59/61/62/63) use a 128-bit buffer where the compiler supports __int128.
  • Less copying — each bit-packing group is read with one bulk read instead
    of per-byte read_ui8, and the pattern length wraps the caller's ByteStream
    instead of copying it into a temporary buffer.

Correctness notes

  • The SIMD gather reads up to 7 bytes past the last packed group, so buffers
    must hold 8 readable bytes of padding after the final group. The RLE decoders
    already zero-pad; the Sprintz block decoders now do too, and the packer tests
    pad their buffers and pass the exact packed length.
  • MSVC has no __int128, so the 128-bit scalar path is guarded by
    __SIZEOF_INT128__ with the portable bit-by-bit loop as the fallback.

Measurements

Unpack micro-benchmark, values/s, -O2 without -mavx2 (runtime dispatch
takes the AVX2 path on the test machine):

bit width before after
14 ~99 M ~547 M
20 ~99 M ~519 M

End-to-end on a wide-table workload (5 features x 4500 measurement columns x
1 month of minute data): this step alone took a single query from 3.14 s to
2.38 s.

Testing

  • Full C++ suite on this branch: 1428 tests pass.
  • The same files also pass a 1440/1440 run under -Denable.asan=ON (Debug) as
    part of a larger branch, including LeakSanitizer.

RLE was the last hot value codec without a batch path: Plain, Gorilla and
TS_2DIFF got read_batch_* in apache#823/apache#898, while RLE still decoded value by value
through read_int and unpacked each bit-packing group with the portable scalar
packer. On a wide table the RLE decoder dominates the read profile.

- add read_batch_int32/int64 and read_batch_float/double: an RLE run becomes a
  std::fill, a bit-packing group becomes one memcpy out of the unpacked buffer.
- unpack four values per step with a SIMD gather+shuffle kernel and dispatch to
  native AVX2 intrinsics at runtime (__builtin_cpu_supports) instead of relying
  on the translation unit being compiled with -mavx2; the simde fallback stays
  for other targets, and the widths that can cross a ninth byte (59/61/62/63)
  use a 128-bit buffer where the compiler has one.
- read each bit-packing group with one bulk read instead of per-byte read_ui8,
  and wrap the caller's stream for the pattern length instead of copying it.
- the SIMD gather reads up to 7 bytes past the last group, so the RLE and
  Sprintz decoders pad their buffers by 8 zero bytes; the packer tests do the
  same and pass the exact packed length.
- guard the 128-bit scalar path with __SIZEOF_INT128__ and keep the portable
  bit loop on MSVC.
A corrupt bit-packing header could size the new bulk read with a wrapped
uint32 product, turn bytes_to_read negative, and write the rest of the
pattern past a 7-byte scratch vector. Validate the bit width, group byte
count and last_bit_packed_num before sizing anything, and reject patterns
whose declared groups do not fit instead of clamping them.

- read_bit_packing_buffer: compute group_bytes in 64-bit, bound it by the
  remaining pattern bytes and mem_alloc's uint32 argument; validate
  bit_width_ (1..32 / 1..64).
- call_read_bit_packing_buffer: require last_bit_packed_num in [1,8],
  bound the value count by INT32_MAX, and clear the count when the group
  load fails; read_batch_* now refuses a non-positive count instead of
  passing it to memcpy.
- read_length_and_bitwidth: reject oversized length prefixes, check the
  bit-width read, and drop the borrowed byte_cache_ view on
  reset()/destroy(); the int64 decoder deletes the old packer per pattern.
- unpack_all_values now decodes the final group from a zero-padded copy so
  callers no longer need the +8 tail; guard zero-width non-advancing loops.
- hoist the simde shuffle mask to a constant byte array, removing the
  thread-safe-static guard from every kernel call.
- Sprintz decoders reject out-of-range block widths.

Add differential batch/scalar tests (wrapped and owning streams, capacities
1..1024, mixed run and bit-packed segments, cross-pattern batches, reset and
reuse), malformed-header rejection tests, Float/Double RLE batch tests, and
exact-size packer tests.
@ColinLeeo
ColinLeeo merged commit 1848341 into apache:develop Nov 9, 2026
49 checks passed
@ColinLeeo
ColinLeeo deleted the perf/rle-simd branch November 9, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant