OnPair

OnPair is a Rust codec for compressing string columns while keeping every row independently accessible. It populates a dictionary of up to 2^16 patterns recurring across the column, then encodes each row as a sequence of indices into it. Decoding just looks up each index and copies out the bytes, so columns decompress at high throughput, and any single row can be read or dropped without touching the ones around it.

Why OnPair

• Fast decompression. The decoder looks up each index and copies out the bytes, with no entropy-decoding stage in the hot path, so decoding sustains high throughput.
• Random access. Each row is a self-contained run of codes, so it decodes on its own, with no block to unpack and no neighbours to rebuild.
• Efficient compression. OnPair reaches ratios competitive with far heavier methods while keeping its encoding pass fast, so strong compression comes at a fraction of the usual cost.
• Compressed-domain search. Evaluate equality, prefix, and substring predicates without decoding rows back to bytes.

OnPair is designed for high-cardinality columns, whose many distinct values tend to share substrings rather than repeat in full. On low-cardinality columns it is better to deduplicate first: encode the column as references to the distinct values, then run OnPair over that set of unique strings. Deduplication removes the value-level repetition, and OnPair compresses the substring redundancy that remains.

Quick start

cargo add onpair

use onpair::{compress, Config};

// Three values in the Arrow layout OnPair takes: one byte buffer plus offsets.
let values = b"catdogbird";
let offsets = [0u32, 3, 6, 10]; // "cat", "dog", "bird"

let column = compress(values, &offsets, Config::default()).unwrap();
let view = column.view();

// A row decompresses to its original value, so a buffer of the longest value fits
// any row.
let longest = 4; // "bird"
let mut buffer = Vec::with_capacity(longest);

let row = view.decompress_row_into(1, buffer.spare_capacity_mut()).unwrap();
assert_eq!(row, b"dog");

Config::default() caps the dictionary at 2^12 entries; raise max_dict_bits for a larger one, up to 2^16.

OnPair is licensed under Apache-2.0.