Skip to content

Render without allocating in the OAR core and OLR - #58

Open
trsonic wants to merge 6 commits into
mainfrom
fix/oar-render-no-alloc
Open

trsonic wants to merge 6 commits into
mainfrom
fix/oar-render-no-alloc

Conversation

@trsonic

@trsonic trsonic commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

For #57: the calls a real-time host makes for every block (gain and position pushes, oar_update_audio_element_data(), oar_render()) allocate and free on every block on main. After these commits they make none outside OBR, from the first block after the elements are added, and the rendered output is bit-identical to main.

  1. oar_render() scratch in the oar_t. The per-group blocks and the renderer output block are kept in the oar_t, sized by oar_add_audio_group() on the configuring thread and zeroed per call; oar_render() returns ck_oar_error_inval instead of allocating if they were too small, which the API does not allow. A configuration with zero output channels or zero samples per channel returns ck_oar_error_inval before touching the scratch; on main it allocated zero-length buffers and, with an element holding data, returned ck_oar_ok while rendering nothing. oar.h no longer lists ck_oar_error_nomem for oar_render().
  2. Metadata pushes. A queue keeps the nodes it pops for its next push (queue_reserve() adds spares). A metadata item (an element's gains and positions, a group's output gain) reserves four nodes and four metadata copies at metadata_item_init() and reuses elapsed copies. A push beyond the spares, or of a ck_param_multiple gain array, allocates as before. An element's gain item is still created on the first gain push for its id, so a host that pushes from its render thread pushes one zero-duration gain per element after adding it (it covers no sample, see Question: 0 duration metadata updates #20).
  3. Object element sub-frames. The constant positions of a sub-frame are filled into a stack oar_metadata_t (metadata_constant_polar_positions_fill() replaces _create(); the libraries' metadata_update() copy the fields they use), and an object element renderer allocates its sub-frame blocks at creation.
  4. OLR. Each track keeps its pending metadata blocks by value in a ring of eight that doubles when a caller queues more, builds its processing block in storage sized for the configuration's samples per frame (a new max_block_samples argument of object_audio_renderer_create()), and calculates gains into the older of two arrays allocated at creation; the tracks are created when OLR opens, a deque reuses its nodes, the per-render channel output is allocated at creation, and the panners and the custom gain calculator use stack arrays bounded by def_max_speakers. OLR no longer keeps every metadata block it received.
  5. Binaural staging block. The binaural element renderer sizes its staging block in add_element() and remove_element() instead of the next add_data(), and render() and apply_gains() return ck_oar_error_nomem while it is not sized for the current elements.
  6. Zero-duration metadata without a warning. metadata_item_elapse() resets a metadata item whose queue held only zero-duration blocks without logging "negative duration": such a queue covers no samples, like an empty queue, for which it already logged nothing. A host that pushes a zero-duration gain per element after adding it (point 2) logged the warning from its render thread, through fprintf(stderr) and localtime(), at a render before that element's next gain push. A queue whose blocks cover part of the rendered samples still warns.

Measured with a C program that runs the per-block sequence through oar.h (48 kHz, 512 samples per channel, one group, a zero-duration gain per element after adding it) and counts allocations on the calling thread with __sanitizer_install_malloc_and_free_hooks() in ASan builds of main and of this branch:

Output Elements Per block, main Per block, this branch First 32 blocks, main First 32 blocks, this branch
stereo 7.1.4 channel-based 7 0 224 0
stereo 3OA scene 7 0 224 0
stereo 8 one-object 198 0 6392 0
stereo 7.1.4, 3OA, 2 one-object 57 0 1846 0
7.1.4 7.1.4 channel-based 7 0 224 0
7.1.4 3OA scene 7 0 224 0
7.1.4 8 one-object 319 0 10228 0
7.1.4 7.1.4, 3OA, 2 one-object 92 0 2775 0
binaural 7.1.4 channel-based 597 590 19105 18880
binaural 3OA scene 31 24 993 768
binaural 8 one-object 5245 5200 163218 161777
binaural 7.1.4, 3OA, 2 one-object 8320 8301 258558 257949

The binaural counts that remain are inside OBR's ObrImpl::Process() and its C API (#28), which this PR does not change. Counts are the same with the limiter on and off.

Over 100,000 blocks of 8 one-object elements on 7.1.4 (Release builds, /usr/bin/time -l), the maximum resident set size is 68.5 MB on main and 7.2 MB on this branch (peak footprint 63.9 MB and 2.6 MB); the run took 17.8 s and 13.9 s (one run each).

Output is bit-identical: an FNV-1a digest of the first 64 rendered blocks matches between main and this branch for all 24 combinations of the outputs and element sets above with the limiter on and off, in Release and in ASan builds.

Tests: new test_render_scratch (silent input after noise renders exact zeros; a group added between renders mixes in exactly; a zero-length configuration returns ck_oar_error_inval), test_metadata_reuse (gains and positions pushed eight blocks ahead, per-sample gain arrays and zero-duration metadata render bit-identically to just-in-time pushes; zero-duration metadata alone renders as without it and logs nothing to stderr, and metadata covering half a block logs the warning), test_olr_block_storage (the metadata ring growing from 8 to 32, storage-backed processing blocks against allocated ones, a renderer destroyed while a track holds a processing block) and test_binaural_add_remove (the block after an element is added or removed equals the first block of a new oar_t). cmake -S . -B build -DOAR_BUILD_EXAMPLES=ON && cmake --build build && ctest --test-dir build -L 'oar|obr': 35 of 35 pass in Release and with AddressSanitizer on macOS arm64 (Apple clang). Each changed file is formatted with the repository's .clang-format.

Not tested: Windows, Linux and the Bazel build (the CI workflows cover them). max_block_samples changes the signature of object_audio_renderer_create(), an internal OLR function; nothing outside src/renderer/olr calls it.

Closes #57

oar_render() allocated a block descriptor per audio group and
(group count + 1) zeroed buffers of output channels x samples per
channel on every call, and freed them before returning. A host calls
oar_render() on its render thread for every block, where allocation
is not bounded in time.

The oar_t now keeps these blocks. oar_add_audio_group(), which runs on
the configuring thread, sizes them for the new group count, and
oar_render() zeroes them on each call. oar_render() does not allocate
them: if they were smaller than the group count needs, which the API
does not allow, it returns ck_oar_error_inval.

A configuration with zero output channels or zero samples per channel
has no scratch; oar_render() returns ck_oar_error_inval for it before
touching the scratch. Before, it allocated zero-length buffers with
calloc(), returned ck_oar_error_nomem where calloc(0) returns NULL, and
otherwise, with an element holding data, returned ck_oar_ok while
rendering nothing.
oar.h no longer lists ck_oar_error_nomem as a result of oar_render(),
and says that output->samples_per_channel is set on success.

test_render_scratch.c: a block of silent input rendered after a block
of noise gives exact zeros; a group added between renders mixes in as
the exact sum of the groups rendered alone; a zero-length
configuration holding an element with data returns
ck_oar_error_inval.
Every gain or position push copied the metadata with metadata_clone()
and queued the copy with queue_push(), which allocated a node: two heap
allocations per push. A host that pushes an element gain, an object
position and a group output gain for every block from its render
thread made two allocations per push on that thread, and freed them
when the metadata elapsed.

A queue now keeps the nodes it pops and reuses them for the next push;
queue_reserve() adds spare nodes ahead of time. A metadata item (an
element's gains and object positions, a group's output gain) is set up
by metadata_item_init(), which reserves four queue nodes and four
metadata copies. metadata_item_push() copies the pushed metadata into
a spare, and metadata_item_elapse() returns elapsed copies to the
spares. A push beyond the spares, or of a ck_param_multiple gain whose
gain array is copied, allocates as before, and a copy that owns a gain
array is freed when it elapses instead of kept.

An element's gain item is still created on the first gain push for its
id. A host that pushes from its render thread can push a zero-duration
gain for each element after adding it; such a gain covers no sample
and elapses at the first render in which an element has data.

test_metadata_reuse.c renders the same input through two oar_t
instances that receive the same metadata at different times and
requires bit-identical output: element gains, group gains and
positions pushed eight blocks ahead (beyond the four spares),
per-sample gain arrays alternating with animated gains three blocks
ahead, and a zero-duration gain and position before the first block.
For each sub-frame of an object element, _sub_frames_apply_positions()
allocated a constant-position copy of the position metadata, an input
sub-block and an output sub-block, and freed all three before
returning; the binaural element renderer allocated the same position
copy for each object element on every render.

The constant positions are written into a stack oar_metadata_t by
metadata_constant_polar_positions_fill(), which replaces
metadata_constant_polar_positions_create(); the renderer libraries'
metadata_update() copy the fields they use and keep no pointer. An
object element renderer allocates its sub-frame input and output
blocks when it is created, sized for a whole block of its channels and
of the output layout's, and renders into them. A block whose
sub-frames do not fit returns ck_oar_error_inval before the first
position reaches the library.

Output is unchanged. test_object_based_rendering and
test_metadata_unit_processing, which renders sub-frames shorter than a
block, pass.
OLR made 17 to 35 heap allocations per object per block, depending on
the layout and the position: a copy of each metadata block and a deque
node, the processing block and its interpolation table, the gain array
of each block, the output of each channel, and inside the gain
calculation the layer gains, the VBAP, DBAP and 2D gain arrays, the
argsort pairs and VBAP's per-region gains. A track also created its
processing channel on its first metadata block, which is the first
render. Every metadata block pushed to a track stayed in a vector that
was never trimmed, so OLR's memory grew with playback time, about 94
blocks per second per object at 512 frames.

A deque now keeps the nodes it pops for its next push, and
deque_reserve() adds spares ahead of time. Each track keeps its
pending metadata blocks by value in a ring of eight that doubles when
a caller queues more. The track builds its processing block in a
processing_block_storage_t, whose interpolation table is sized for
max_block_samples, a new argument of object_audio_renderer_create()
that OLR passes as the configuration's samples per frame; a longer
block grows the table. The interpreter calculates each block's gains
into the older of two arrays allocated at creation; the processing
block that used it has been dropped by then, because a track holds
one processing block at a time. object_audio_renderer_add_tracks()
creates the track channels when OLR opens, and the renderer allocates
its per-channel output at creation. A track frees its queued
processing block before the interpreter that holds the block's
storage.

The layerwise, VBAP and DBAP panners and the custom gain calculator
use stack arrays bounded by def_max_speakers (24) instead of heap
arrays and vectors of heap pointers, and nc_argsort() sorts up to 64
values in a stack buffer. interp_gains_create() and
fixed_gains_create() still allocate, for callers outside a track.

test_olr_block_storage.c queues twenty metadata blocks before the
first render, so the ring grows from 8 to 32, and requires the output
of one block queued per render; it requires interpolated and fixed
gains built in storage, including blocks longer than its table, to
process as the allocated blocks do; and it destroys a renderer whose
track still holds a processing block.
The binaural element renderer stages every element's input in one
block. add_element() and remove_element() only marked it out of date,
and the next add_data() call, which a host makes on its render thread,
allocated it (calloc) or grew it (realloc).

_update_staging_block() now sizes the block in add_element() and
remove_element(), on the configuring thread, keeping the data of the
existing channels and zeroing the added ones; add_data() calls it only
if that failed. While the block is not sized for the current elements,
render() and apply_gains() return ck_oar_error_nomem instead of reading
or writing past its end. The resize compares the block's own channel
count with the elements' instead of the channel count of the block
being added.

test_binaural_add_remove.c renders the block after an element is added
(with the existing element's input fed before the add) and after an
element is removed, and requires it to equal the first block of a new
oar_t with the same elements.
metadata_item_elapse() warned "negative duration" whenever the
samples elapsed exceeded the queued duration. A queue holding only
zero-duration blocks covers no samples, like an empty queue, for which
the function returns without a warning, but it warned for each such
item on the next render with data. A host that pushes a zero-duration
gain per element after adding it, so that the element's gain item
exists before rendering starts, logged the warning from its render
thread through fprintf(stderr) and localtime() whenever an element's
queue held only that block at a render: for example when it renders a
new instance's first blocks with silent input and no metadata, or
renders while one element receives neither data nor a gain. The item
now resets silently when its queue covered no samples; a queue whose
blocks cover part of the rendered samples still warns. Output is
unchanged. The warning prints its uint32_t fields with %u.

test_metadata_reuse.c captures stderr around a render in
render_capturing_stderr(). TC4 renders blocks after only zero-duration
metadata, requires the output of blocks rendered without it, and
requires no warning; it fails without the change (four warnings: the
group gain, both element gains and the object positions). TC5 renders
a block after metadata covering half of it and requires the warning,
so TC4 cannot pass because the capture or the warning is missing; it
fails if the warning is never logged.

@jingbo-marquis jingbo-marquis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The per-block render path allocates and frees on the render thread, and OLR's memory grows with playback time

3 participants