Add LengthPrefixedAccessUnitParser for length-prefixed H.264/H.265 by haffi96 · Pull Request #301 · livekit/client-sdk-cpp

haffi96 · 2026-10-27T09:15:06Z

EncodedVideoSource takes Annex-B access units, but hardware encoder bridges can commonly deliver 4-byte length-prefixed NAL units (avc3 / hev1). This adds a parser that turns such a stream, in any chunking (e.g. straight from a TCP socket), into ready-to-capture EncodedVideoSource::Frames.

Converts to Annex-B in place during the single copy out of its buffer (~2 µs per 1080p access unit).
Sets is_keyframe with the same rules as livekit-capture (H.265 needs VPS/SPS/PPS with the IDR).
single_slice_flush completes an access unit as soon as its slice arrives instead of when the next one begins, which otherwise holds every picture for a frame interval.
frame_metadata_from_sei fills Frame::metadata from an LKTS packet-trailer SEI, the convention used by the LiveKit Go SDK.

LengthPrefixedAccessUnitParser::Options options;
options.single_slice_flush = true;
LengthPrefixedAccessUnitParser parser(VideoCodec::H265, options);
parser.push(buf, n);
while (auto frame = parser.next()) {
  source->captureFrame(*frame);
}

Mirrors LengthPrefixedAccessUnitParser in livekit/rust-sdks#1579. That PR also publishes pre-encoded video as a single layer, which matters for C++ because TrackPublishOptions::simulcast defaults to on.

Testing

9 new unit tests (LengthPrefixedAccessUnitParser*), including chunked-vs-one-shot equivalence and emulation-prevention handling in the SEI.
Manual: published a 1080p30 hev1 stream from GStreamer over TCP; steady-state publish latency p50 0.83 ms.
Unrelated: AudioProcessingModuleTest.AGCAttenuatesLoudSpeech fails locally both with and without this change.

🤖 Generated with Claude Code

EncodedVideoSource takes Annex-B access units, but hardware encoder bridges commonly deliver 4-byte length-prefixed NAL units (avc3/hev1). The parser accepts any chunking of such a stream and returns EncodedVideoSource::Frame values: Annex-B (converted in place during the single copy), keyframe flag, and optional packet-trailer metadata from an LKTS SEI (the LiveKit Go SDK convention). single_slice_flush completes an access unit as soon as its slice arrives instead of when the next one begins, which otherwise holds every picture for a frame interval. Mirrors livekit-capture's LengthPrefixedAccessUnitParser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

devin-ai-integration

Devin Review found 5 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-10-27T09:17:24Z

chunking test

+  if (nals_.empty()) {
+    return std::nullopt;
+  }
+  return takeAccessUnit(nals_.size());


🟡 Trailing delimiters become picture frames

When a stream ends with an AUD, flush() returns that AUD as a frame without a slice. Passing it to captureFrame submits a pictureless access unit.

Learn more

An access unit is a group of NALs containing a coded picture. The parser leaves leading AUDs and parameter sets buffered until a slice arrives, but the final flush() returns the buffered group even if no slice arrived. The added chunking test explicitly expects a third output consisting only of a trailing AUD. captureFrame sends this frame to the encoder as if it held a picture.

Example: For [slice A, AUD], next() emits slice A; flush() then emits an AUD-only frame, rather than ending with no more frames.

Recommended fix: Have flush() discard or otherwise handle trailing non-VCL NALs without emitting a frame, and test stream endings after AUD and parameter sets.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-27T09:17:25Z

+    if (options_.single_slice_flush) {
+      // Scanned NALs are complete, so the first slice holds the picture.
+      if (vcl) {
+        return i + 1;
+      }


🟡 Suffix SEI moves to next picture

With single_slice_flush, next() emits an H.265 slice before its suffix SEI arrives. The suffix SEI then joins the following picture, producing wrongly grouped access units.

Learn more

An H.265 suffix SEI is a NAL unit associated with the picture whose slice precedes it. In single-slice mode, the parser completes a picture as soon as that slice is complete. Later suffix SEIs then remain in pending_ and are prepended to the next slice when another picture arrives. This breaks access-unit grouping even though the encoder uses only one slice per picture.

Example: Given length-prefixed [slice A, suffix SEI A, slice B], the parser emits [slice A] and [suffix SEI A, slice B] rather than keeping the suffix with A.

Recommended fix: Define and implement a strategy for post-slice NALs in single_slice_flush mode, or explicitly reject that mode for streams that contain suffix SEIs; ensure the resulting frame boundaries follow the H.265 access-unit ordering.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-27T09:17:27Z

captureFrame

+  if (const auto split = splitIndex(); split && *split > 0) {
+    return takeAccessUnit(*split);
+  }
+  if (pending_.size() > kMaxPendingAccessUnitBytes) {
+    throw std::runtime_error("LengthPrefixedAccessUnitParser: access unit exceeds maximum buffered size");


🟡 Complete oversized frames bypass the limit

When a complete access unit exceeds 32 MiB and another picture begins, next() emits it instead of throwing. The size check runs only when no boundary exists.

Learn more

The parser documents a 32 MiB upper bound on access units. A detected boundary makes next() return through takeAccessUnit() before checking the pending-buffer size, so a complete oversized frame is accepted. The 4-byte length prefix can represent a 33 MiB NAL, and captureFrame accepts frames up to 64 MiB, so this is an observable API-contract difference.

Example: Push a 33 MiB H.264 slice followed by the first slice of another picture; next() returns the 33 MiB access unit rather than raising the documented size error.

Recommended fix: Check the length of the access unit selected by splitIndex() before calling takeAccessUnit(). Keep the existing pending-buffer guard for incomplete frames.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-27T09:17:28Z

public API documentation rules

+  struct Options {
+    /// Size of each big-endian NAL length prefix, 1 to 4 bytes
+    /// (`lengthSizeMinusOne + 1` from `avcC`/`hvcC`; 4 for most encoders).
+    std::uint8_t nal_length_size = 4;
+    /// Complete each access unit as soon as its first slice arrives, instead
+    /// of when the next access unit begins.
+    ///
+    /// The end of an access unit is otherwise only known from the NAL that
+    /// starts the next one, which holds every picture for a full frame
+    /// interval unless the encoder terminates access units with an AUD.
+    /// Enable this only for streams with one slice per picture, which is
+    /// what hardware encoders typically emit.
+    bool single_slice_flush = false;
+    /// Fill Frame::metadata from an LKTS `user_data_unregistered` SEI ahead
+    /// of the slice (the packet-trailer convention of the LiveKit Go SDK).
+    bool frame_metadata_from_sei = false;
+  };
+
+  /// @throws std::invalid_argument if the codec is not H.264 or H.265, or
+  ///         nal_length_size is not 1 to 4.
+  explicit LengthPrefixedAccessUnitParser(VideoCodec codec);
+  /// @throws std::invalid_argument if the codec is not H.264 or H.265, or
+  ///         nal_length_size is not 1 to 4.
+  LengthPrefixedAccessUnitParser(VideoCodec codec, Options options);
+
+  /// @brief Appends stream bytes.
+  void push(const std::uint8_t* data, std::size_t size);
+
+  /// @brief Returns the next complete access unit, if one is buffered.
+  ///
+  /// Frame::width/height are left at zero (the source resolution) and
+  /// Frame::timestamp_us at zero (capture time).
+  /// @throws std::runtime_error on malformed input, or when an access unit
+  ///         exceeds 32 MiB.
+  std::optional<EncodedVideoSource::Frame> next();
+
+  /// @brief Returns the remaining buffered NAL units as the final access unit.
+  /// Call repeatedly until it returns nullopt.
+  /// @throws std::runtime_error on a truncated final NAL unit.
+  std::optional<EncodedVideoSource::Frame> flush();


🟡 Public parser omits required API tags

The new Options struct lacks @brief; constructors lack @param, and next() and flush() lack @return. These omissions violate the repository's public API documentation rules.

Learn more

The repository's public API documentation rules require @brief for structs and methods, @param for parameters, and @return for non-void methods. The new Options struct has no @brief. Both constructors lack @brief and @param, push lacks @param, and next/flush lack @return.

Example: A consumer viewing the generated reference for push cannot find the required documentation for its data or size arguments.

Recommended fix: Add the missing Doxygen tags to each new public declaration, including both constructor overloads and the push arguments.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-27T09:17:29Z

+void LengthPrefixedAccessUnitParser::push(const std::uint8_t* data, std::size_t size) {
+  pending_.insert(pending_.end(), data, data + size);
+}


🟨 Stream input bypasses parser memory cap

When a stream supplies an oversized chunk, push() allocates it before next() checks the 32 MiB limit. Untrusted video input can exhaust process memory before rejection.

Was this helpful? React with 👍 or 👎 to provide feedback.

haffi96 · 2026-10-27T09:42:58Z

The main motivation here is to enable the H265 pre-encoded ingest path with length prefixed NAL units more or less trying to mirror Livekit Go SDK pre-encoded publish path

See rust PR livekit/rust-sdks#1579

haffi96 requested review from a team and xianshijing-lk as code owners October 27, 2026 09:15

devin-ai-integration Bot reviewed Oct 27, 2026

View reviewed changes

haffi96 marked this pull request as draft October 27, 2026 09:20

Navigation Menu

Repository navigation

Add LengthPrefixedAccessUnitParser for length-prefixed H.264/H.265 - #301

Add LengthPrefixedAccessUnitParser for length-prefixed H.264/H.265#301
haffi96 wants to merge 1 commit into
livekit:mainfrom
haffi96:feat/h265-pre-encoded-length-prefixed

haffi96 commented Oct 27, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Uh oh!

haffi96 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

haffi96 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

haffi96 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

haffi96 commented Oct 27, 2026 •

edited

Loading