Everything the library does, by example. The wire format itself is defined by STANDARD.md — a verbatim vendored copy of the specification in mas-bandwidth/serialize, which CI checks for drift; this document teaches the Java surface that speaks it.
import serialize.*;The family's defining pattern: write, read and measure share a single
serialize function through the BitStream interface. Every operation
returns a bool, values travel in holder cells — primitive-specialized
IntRef, LongRef, BoolRef, FloatRef and DoubleRef (Java has no
ref parameters, and a specialized cell keeps the hot path free of
boxing), plus Ref<T> for String and the 128-bit pair — and the
stream direction decides whether the holder is consumed or filled.
final class Player
{
final IntRef health = new IntRef();
final BoolRef alive = new BoolRef();
final FloatRef heading = new FloatRef();
}
static boolean serializePlayer( BitStream stream, Player player )
{
return stream.serializeInt( player.health, 0, 100 )
&& stream.serializeBool( player.alive )
&& stream.serializeFloat( player.heading );
}
// write
byte[] buffer = new byte[64]; // length a multiple of 8
WriteStream writer = new WriteStream( buffer, buffer.length );
serializePlayer( writer, player ); // -> true
writer.flush(); // ALWAYS flush before touching the bytes
int packetBytes = (int) writer.getBytesProcessed(); // 5 bytes: 7 + 1 + 32 bits
// read
ReadStream reader = new ReadStream( buffer, packetBytes );
Player decoded = new Player();
serializePlayer( reader, decoded ); // -> true
// decoded.health.value == 87
// measure
MeasureStream measure = new MeasureStream();
serializePlayer( measure, player ); // -> true
// measure.getBitsProcessed() == 40WriteStream needs a buffer whose size is a multiple of 8 (the writer
stores qwords to memory); the packet to send is the first
getBytesProcessed() bytes. MeasureStream touches no memory at all —
it prices a message so you can size buffers; its bound is conservative
(see Measuring).
The read allocation contract, matching the C++ reference: the byte
array handed to ReadStream (or BitReader) must extend at least 8
bytes past the end of the data being read, because the reader loads
8-byte windows at byte granularity. The bytes past the end are loaded
but never interpreted — they can never influence a decoded value or an
accept/reject decision. Reading a packet out of the buffer it was
written into, as above, satisfies the contract for free; a packet
arriving from a socket just needs to land in an array 8 bytes longer
than the largest packet.
All three streams expose getBitsProcessed(), getBytesProcessed(),
isWriting() / isReading(), getAlignBits(), and reset(...) for
allocation-free reuse — point a stream at a buffer again and all state
clears, no new stream object needed.
The check model is the family standard's ("Writes assume trusted data"): the caller is responsible for well-formed writes, and reads validate everything, because the wire is a trust boundary.
On the read side, every failure — a truncated read, a value outside its
range, nonzero alignment padding, a malformed string — returns false,
and hostile bytes never throw.
A failed read is terminal. Nothing after the failing operation has a
defined position, so ReadStream latches: the first refusal sets the
failure flag, and every later read on that stream returns false
without consuming a bit or writing a destination — a zero-bit read
included. isFailed() reports the latch; reset(...) points the stream
at data again and clears it, and nothing else does.
A refused read leaves its destination unwritten. When a read of a
scalar fails, the holder cell holds exactly what it held before the
call, so a caller that trusts the cell over the return code cannot
proceed on a value the stream never carried. Two limits: a read into a
caller-owned buffer — serializeBytes, serializeString,
serializeWideString — leaves that buffer's contents unspecified
after a refusal, and a composite read may leave earlier members written,
because it is a sequence of primitive reads and each one carries the
rule alone.
byte[] data = new byte[16]; // 1 byte of data + the 8-byte slack
ReadStream r = new ReadStream( data, 1 ); // 8 bits of data
r.serializeBits( v, 32 ); // -> false: past the end
r.serializeBits( v, 8 ); // -> false: the stream has failed, v untouched
r.isFailed(); // -> true
r.reset( data, 1 ); // point at data again, the latch cleared
r.serializeBits( v, 8 ); // -> true
// an offset smuggled into the bit headroom of a range is refused
ReadStream r2 = new ReadStream( hostile, 1 ); // 8 bits carrying 255
r2.serializeInt( ranged, 0, 200 ); // -> false: 255 > 200One rule follows from clean refusal: check the result of any serialized value that controls a loop before the loop uses it, or a truncated packet spins the loop on garbage.
On the write side, WriteStream and MeasureStream operations always
return true — caller contracts (bit counts in range, values within
their declared ranges, min ≤ max, well-formed string payloads, buffers
that fit) are assert statements, the Java form of the family's debug
asserts:
- Checked (
java -ea, howmake testruns): contract violations throwAssertionErrorat the call site. Develop and test here. - Release (a plain
javainvocation — assertions are disabled by default): asserts compile to nothing at runtime, exactly as a C++ release build compilesserialize_assertto nothing. The caller is trusted; misuse produces garbage on the wire (which conforming readers refuse), never memory unsafety — the JVM's own bounds checks backstop the trusted path.
The wire for conforming writes is byte identical in both modes, and so
is every refusal: the read side's obligations are checks, never asserts.
make test-release runs the whole suite in the release shape to prove
it.
serializeBits moves the low bits of an IntRef, 1 to 32.
serializeBits64 is its LongRef twin, 1 to 64 bits. Values wider than
32 bits go low 32-bit dword first — the family's group rule.
WriteStream w = new WriteStream( buffer, buffer.length );
w.serializeBits( new IntRef( 5 ), 3 ); // 3 bits on the wire
w.serializeBits( new IntRef( 0xdeadbeef ), 32 ); // full width
w.serializeBits64( new LongRef( 0x123456789abcdef0L ), 64 ); // low 32-bit group first
w.serializeAlign(); // zero-pads to the byte boundary
w.flush();serializeAlign writes zero bits up to the next byte boundary (nothing
if already aligned); the reader verifies the padding is zero and refuses
otherwise.
serializeInt(ref, min, max) is the format's defining operation: the
value rides as an offset from min in exactly
SerializeUtil.bitsRequired(min, max) bits. Both sides must state the
same range — the range is part of the message format, not the wire.
w.serializeInt( new IntRef( -37 ), -100, 100 ); // 8 bits
w.serializeInt( new IntRef( 7 ), 7, 7 ); // degenerate range: ZERO bitsReads refuse values smuggled into the bit headroom of a range (an offset
above max - min fails the read — reject, never clamp).
serializeInt64 is the same operation in the long domain, with offsets
computed in unsigned arithmetic so ranges wider than 2^63 are exact.
serializeInt128 extends it to 128 bits on the Int128Value pair,
written in 32-bit groups least significant first; where the range fits
64 bits the bytes are identical to serializeInt64. min <= max is the
legal relation on every ranged width: a degenerate min == max costs
zero bits at 128 bits exactly as it does at 32, with nothing on the
wire, nothing consumed, and the value taken from min.
w.serializeInt64( new LongRef( -5000000000L ), -5000000000L, 5000000000L ); // 34 bits
w.serializeInt128( new Ref<>( Int128Value.fromLong( -1 ) ),
Int128Value.MIN_VALUE, Int128Value.MAX_VALUE ); // 128 bitsSerializeUtil.bitsRequired, bitsRequired64 and bitsRequired128
price a range when designing a message format. The subtraction wraps in
the unsigned domain, so the full int32/int64 ranges and ranges wider
than 2^63/2^127 report exact widths:
SerializeUtil.bitsRequired( 0, 200 ); // 8: the cost of serializeInt over [-100, +100]
SerializeUtil.bitsRequired64( 0, 5000000000L ); // 33
SerializeUtil.bitsRequired128( UInt128Value.ZERO,
UInt128Value.fromUnsignedLong( 1 ).shiftLeft( 100 ) ); // 101SerializeUtil also carries zig-zag conversion (signedToUnsigned /
unsignedToSigned) for callers that want small-magnitude signed values
in an unsigned encoding.
Fixed-width conveniences. The 8/16/32-bit helpers ride IntRef, 64 bits
rides LongRef, and serializeUint128 is always 128 bits on the
UInt128Value pair, low 64-bit half first. serializeBool is one bit.
w.serializeUint8( new IntRef( 0x7f ) );
w.serializeUint16( new IntRef( 0x1234 ) );
w.serializeUint32( new IntRef( 0x12345678 ) );
w.serializeUint64( new LongRef( 0xfedcba9876543210L ) ); // unsigned, bit transparent in long
w.serializeUint128( new Ref<>( UInt128Value.fromUnsignedLong( 1 ).shiftLeft( 100 ) ) );
w.serializeBool( new BoolRef( true ) );Unsigned-long discipline: Java has no unsigned 64-bit type, so
unsigned wire values ride bit-transparently in long — arithmetic wraps
two's complement, which is exactly the unsigned-domain wrap the format's
offset encodings rely on. Where ordering or printing matters, go
through the JDK's unsigned statics: Long.compareUnsigned,
Long.toUnsignedString, Long.divideUnsigned. The same discipline
applies to int for 32-bit wire values (Integer.compareUnsigned and
friends).
Int128Value / UInt128Value are immutable two's complement
(hi, lo) pairs with the arithmetic, shift and comparison operations
the serialize surface needs. Int128Value.fromLong sign-extends;
UInt128Value.fromUnsignedLong zero-extends; toUnsigned() /
fromUnsigned reinterpret bit-transparently.
serializeFloat (32 bits) and serializeDouble (64 bits) reproduce the
transmitted pattern exactly — every pattern is legal on the wire: NaNs
with any payload, signaling NaNs, infinities, negative zero, denormals.
The library moves patterns with floatToRawIntBits /
intBitsToFloat, so nothing is quieted or canonicalized in either
direction.
w.serializeFloat( new FloatRef( 3.1415926f ) );
w.serializeDouble( new DoubleRef( 1.0 / 3.0 ) );
w.serializeFloat( new FloatRef( -0.0f ) ); // -0 round trips as -0serializeCompressedFloat(ref, min, max, resolution) quantizes into a
declared range at a resolution — the one lossy operation. The
declaration is part of the message format. The arithmetic is float32
with the standard's two roundings on each side — Java's float
arithmetic is strict IEEE-754 with no contraction, so the wire matches
the family's discriminating vectors bit-exactly.
w.serializeCompressedFloat( new FloatRef( 5.0f ), 0.0f, 10.0f, 0.01f ); // 10 bits
// reading it back yields exactly 5.0f: the value sits on a quantum.
// off-quantum values come back within the resolution; re-encoding a
// decoded value is byte-identical (the round trip is idempotent).Finite values outside [min, max] clamp on write; writing a non-finite
value is a contract violation (asserted under -ea, clamped in
release); reads refuse integers smuggled above the quantization ceiling.
SerializeUtil.compressedFloatMaxIntegerValue exposes the declaration's
step count for callers pricing a format.
serializeBytes(data, bytes) aligns to the byte boundary (the alignment
is part of the format, padding verified on read) and then bulk-copies
bytes bytes. The count is never transmitted: both sides agree by
passing the same count. On read, the array you pass is filled in place.
byte[] blob = new byte[] { (byte) 0xde, (byte) 0xad, (byte) 0xbe, (byte) 0xef };
w.serializeBytes( blob, blob.length );
// ...
byte[] out = new byte[4];
r.serializeBytes( out, out.length ); // out now holds the bytesA zero-length write still performs (and verifies) the align.
serializeString(ref, bufferSize) sends the UTF-8 byte length as
serializeInt(length, 0, bufferSize - 1), then the payload as
serializeBytes (which aligns). bufferSize is part of the message
format — the same string against different buffer sizes produces
different bytes — and the payload must fit bufferSize - 1 bytes.
w.serializeString( new Ref<>( "golden" ), 16 );
Ref<String> s = new Ref<>();
r.serializeString( s, 16 ); // s.value.equals( "golden" )Reads validate the payload in every mode: malformed UTF-8 (overlongs,
surrogate code points, values above U+10FFFF, truncated sequences, stray
continuations) and interior NULs fail the read. A Java string carrying
an unpaired surrogate is a writer contract violation (asserted under
-ea): the UTF-8 encoder would replace it, so it cannot reach the wire
faithfully.
serializeWideString(ref, bufferSize) sends the unit count, then one
32-bit group per UTF-16 code unit — never a code point — with no
alignment anywhere: the one place the wide path deliberately differs
from its narrow counterpart. A Java string is a sequence of UTF-16
code units, so astral characters are two groups on the wire, exactly as
the family's 2-byte-wchar_t ports split them. bufferSize counts wide
characters.
w.serializeWideString( new Ref<>( "😀A" ), 8 ); // 3 code units: 99 bitsUnpaired surrogates in a written string are a contract violation
(asserted under -ea — the wide wire cannot carry ill-formed UTF-16,
because conforming readers refuse it). Reads refuse groups above 0xFFFF,
interior NUL groups, and unpaired, misordered or dangling surrogates.
serializeIntRelative(previous, ref) prices strictly increasing
sequences — sequence numbers, ack chains. The domain is 0 to
2^31 - 1 inclusive, and both previous and current lie in it.
current > previous always, no wrapping: a caller with a wrapping
counter unwraps it before serializing. A difference of 1 costs a single
bit; small differences ride payload tiers of 5/8/13/18/23 bits; past the
ladder, six zero flags carry current itself as 32 raw bits.
The reader reconstructs current in a width that cannot wrap, in every
tier, and refuses the read unless the result lies in the domain and
strictly exceeds previous — the absolute tier's 32 raw bits are
unsigned, so a value with the top bit set is outside the domain and is
refused. A refused read is terminal and leaves the holder cell
untouched.
w.serializeIntRelative( 100, new IntRef( 101 ) ); // 1 bit
w.serializeIntRelative( 100, new IntRef( 2100 ) ); // a mid-ladder tier
// read side: pass the same previous, get current back
r.serializeIntRelative( 100, seq ); // seq.value == 101previous is caller state, not wire: both sides already know it, and it
never arrives off the wire. Writing current <= previous, or a
previous outside the domain, is a contract violation, asserted under
-ea.
serializeFixed(ref, integerBits, fractionBits, minUnits, maxUnits)
carries Q-format fixed point. ref.value is the raw scaled integer
— the real value times 2^fractionBits — in a LongRef, with storage
of exactly integerBits + fractionBits bits (8, 16, 32 or 64; the sign
bit counts toward integerBits). The bounds are in whole units,
part of the message format. serializeFixed128 is the 128-bit storage
counterpart: integerBits + fractionBits must equal 128, and the value
is a Ref<Int128Value>.
// -3.25 in Q8.8 over [-100, +100] whole units: raw is -3.25 * 256 = -832
w.serializeFixed( new LongRef( -832 ), 8, 8, -100, 100 ); // 16 bits
// 1234.5 in Q16.16 over [-2000, +2000]
w.serializeFixed( new LongRef( 1234L * 65536 + 32768 ), 16, 16, -2000, 2000 );
// 12345.5 in Q48.16 over [-100000, +100000]: 64-bit storage
w.serializeFixed( new LongRef( 12345L * 65536 + 32768 ), 48, 16, -100000, 100000 ); // 34 bits
// Q64.64 over [-1000, +1000] whole units: 128-bit storage
Ref<Int128Value> q6464 = new Ref<>( Int128Value.fromLong( 1 ).shiftLeft( 64 ) ); // exactly 1.0
w.serializeFixed128( q6464, 64, 64, -1000, 1000 );The wire is the offset from min << fractionBits in exactly the bit
length of the raw range — byte identical to serializeInt64 of the raw
value wherever storage fits 64 bits — and the round trip is exact:
no quantization, unlike the compressed float. A degenerate min == max
range costs zero bits on every storage width. Signedness never reaches
the wire: for the same bounds, signed and unsigned storage produce
identical bytes. Reads refuse raw values smuggled into the bit headroom;
an invalid declaration is caller misuse, asserted under -ea.
serializeObject(serializer) runs a nested object's own serialize
function inline. It contributes no bytes of its own — no framing, no
length prefix and no alignment is inserted around it — so the nested
operations land at exactly the position the call sits at, and nesting a
run of fields changes nothing about the bytes.
Serializer is a one-method interface, so a message type implements it
directly and a small nested field can be a lambda:
final class Transform implements Serializer
{
final FloatRef x = new FloatRef();
final FloatRef y = new FloatRef();
@Override public boolean serialize( BitStream stream )
{
return stream.serializeFloat( x ) && stream.serializeFloat( y );
}
}
// the same three fields, nested and flat, produce the same bytes
stream.serializeInt( health, 0, 100 ) &&
stream.serializeObject( transform );
stream.serializeInt( health, 0, 100 ) &&
stream.serializeFloat( transform.x ) && stream.serializeFloat( transform.y );A refusal inside the nesting propagates out of it, and a read stream consults its failure state first, so a nested object on a stream that has already failed refuses without invoking the object at all.
MeasureStream prices a message without a buffer. For everything except
alignment it is exact; any operation that aligns (serializeAlign,
serializeBytes, serializeString) charges the worst case — 7 bits of
padding — because the measure stream cannot know what alignment the
field will land on inside your message. The guarantee is a bound, never
equality:
measure.getBitsProcessed() >= writer.getBitsProcessed(); // always trueSize buffers from the measured bound (rounded up to a multiple of 8
bytes for WriteStream).
BitWriter and BitReader are the streams' engine — the family wire in
branchless 64-bit window loads — and are public for code that wants raw
bitpacking without the serialize surface or its checks:
BitWriter bw = new BitWriter( buffer, buffer.length );
bw.writeBits( 5, 3 );
bw.writeAlign();
bw.writeBytes( new byte[] { 1, 2, 3 }, 3 );
bw.flushBits();
BitReader br = new BitReader( buffer, (int) bw.getBytesWritten() );
br.readBits( 3 ); // 5
br.readAlign(); // true: padding was zero
byte[] out = new byte[3];
br.readBytes( out, 3 ); // out is now { 1, 2, 3 }The reader carries the same allocation contract as ReadStream: 8 bytes
of slack past the data.
The same values produce the same bytes in every family implementation.
This is not aspiration but pinned fact: the test suite runs every vector
in conformance/ — the family's shared corpus, vendored
from mas-bandwidth/serialize and checked for drift by CI — and carries
the family's golden vectors: the golden wire message covering every
operation class byte for byte, the discriminating float vectors, the
string and wide-string pins, every relative-integer tier, and the fixed
point shapes at every group count, all minted from the canonical C++
reference's own output. If your message serializes with
the same declarations on both ends, a stream written by any family
implementation reads in any other.
Two doctrines worth knowing at the edges:
- Trailing bits: writers zero the unused bits of the final byte; readers never reject a stream for their contents.
- Past-end data: the slack bytes past the end of the data are loaded but never interpreted — they cannot influence a decoded value or an accept/reject decision.