H.264 decode speed on x86-64 #22

Goal

CoreAV's H.264 decoder is no slower than FFmpeg on x86-64: the median per-stream time ratio (CoreAV over FFmpeg) is at most 1.00 overall and in every resolution class (sd, 720p, 1080p, 4k), at one thread and at eight threads, on the held-out streams.

Current state

• Opt-in HEVC interpolation reuses the existing residual workspace for temporary rows (perf(hevc): reuse interpolation row scratch #539, merged as 3951933e). Complete automatic and forced-scalar conformance passes; filtered samples, validation, allocation, scratch accounting and defaults are unchanged.

• Opt-in HEVC intra-reference gathering reuses availability within each four-by-four luma cell during one component call (perf(hevc): reuse intra neighbour availability #538, merged as ba38f93e). Complete automatic and forced-scalar conformance passes; decoded output, per-sample bounds, slice and tile admission, constrained intra rules and statistics are unchanged.

• Opt-in HEVC prediction skips the unused comparison predictor unless the current component has nondefault weights (perf(hevc): skip unused comparison predictions #536, merged as 68cbc803). Complete automatic and forced-scalar conformance passes; decoded output, destination validation, weighting, statistics and allocation are unchanged.

• Opt-in HEVC Main10 interpolation now borrows checked interior reference rows too (perf(hevc): borrow Main10 reference rows #532, merged as ef6821e2). Complete automatic and forced-scalar conformance passes; filter arithmetic, shifts, edge replication, validation, memory accounting and defaults are unchanged.

• Opt-in HEVC eight-bit interpolation borrows checked interior reference rows instead of gathering copies (perf(hevc): borrow interior reference rows #528, merged as de671521). Complete automatic and forced-scalar conformance passes. Filter arithmetic, edge replication, scratch accounting, ten-bit behavior and defaults are unchanged.

• Opt-in HEVC eight-bit residual reconstruction borrows complete active rows and preserves full-domain clipping, validation errors and output padding (perf(hevc): streamline eight-bit reconstruction #502, merged as 7a55d8ab). Complete automatic and forced-scalar conformance passes; ten-bit reconstruction, SIMD and scratch accounting are unchanged.

• Opt-in HEVC uni, bi and weighted prediction combine validated rows with signed i32 arithmetic (perf(hevc): combine predictor rows #520, merged as 7616faff). Full automatic and forced-scalar conformance passes; decoded samples, errors, defaults and isolation stay unchanged. No new SIMD is added.

• Opt-in native HEVC now decodes eligible wavefront rows concurrently through the existing thread setting (perf(hevc): decode wavefront rows in parallel #495, merged as 90900454). Complete conformance preserves identical normalized records for 144 streams and 8,195 pictures at 1, 2, 4 and 8 threads. Whole-picture filtering, defaults and syntax admission are unchanged. Frame scheduling and tiles remain follow-up work.

• Opt-in HEVC coefficient scans use compile-time forward and inverse lookup tables with checked access (perf(hevc): precompute coefficient scans #499, merged as 81271b80). Decoded output, syntax admission and CABAC arithmetic are unchanged.

• Opt-in HEVC coefficient-level decoding skips zero positions while preserving bin order, output, errors and budgets (perf(hevc): skip zero coefficient positions #515, merged as ae8e267b).

• Packed native H.264 output storage is reused in the CLI and transcode (perf(decode): reuse frame buffers #458, merged as 79fc6131). Raw writers consume CPU pictures and stream their planes without clone or staging buffers. Raw output hashes and AV1 packet identities are unchanged. Decoder kernels, release allocators, isolation, build defaults and the official decoder snapshot are unchanged.

• The Linux release binaries (x86-64 and arm64 musl, default and full builds) allocate through mimalloc 3.3.2 instead of musl's allocator (perf(cli): use mimalloc in Linux musl binaries #466, merged as 1ed05684), which removes most of musl's mmap/munmap and page-fault cost in decoding. Output and other platforms are unchanged, and the binaries carry mimalloc's licence (release.md, "Allocator"). The official decoder snapshot does not include it.

The official snapshot of 2026-10-29 against FFmpeg 9.0.2 (snapshot-361cd29f), held-out streams, decoding with output discarded:

• Eight threads: 0.828 overall (95% interval 0.764 to 0.857), CoreAV faster on 62 of 64 streams; every class is below 1.00 (sd 0.874, 720p 0.861, 1080p 0.819, 4k 0.656).
• One thread: 0.958 overall (0.923 to 0.985), faster on 45 of 64; 720p is the one class above the bar at 1.001 (0.980 to 1.019), and 1080p is 0.974 with an interval reaching 1.033.

Bars and method: docs/design/h264.md ("Bars", "Measurement method").

• Round tooling (fix(h264): refuse parity runs with failed streams #34): scripts/h264_parity.py exits with status 3 when a stream it was asked to time fails, and scripts/h264_parity_bars.py is the one analyzer for a round's bars, refusing reports that miss a stream or hold a time or ratio that is not a finite positive number, and bars files with a subset named per_stream (fix(h264): refuse a bars subset named per_stream #45).

• Bundle 3b (perf(h264): bundle 3b cuts one-thread time 1.1% #58), merged after the snapshot: one thread k/base 0.9886 over the 32 training streams and 0.9934 (0.9899 to 0.9958) over the 64 held-out streams, faster on 49 of 64, on the 2.90 GHz class its bars named; eight threads 0.9984 discard and 0.9994 write; output unchanged on 96 of 96 streams (bundle3b/gating). The snapshot above does not include it yet.

• One-thread host (docs(h264): fixed-frequency one-thread host #55): one-thread arms of rounds whose bars are committed from now on run on the fixed-frequency Compute Engine class (c4-standard-8, one thread a core, turboMode=ALL_CORE_MAX, Xeon Platinum 8581C, --one-thread-class --turbo all-core-max). Its A/A and null-arm calibration met every pre-registered bar (gcp-1t-fixed). One-thread bars measured on earlier classes do not transfer to it.

• Several sample descriptions (fix(decode): follow MP4 sample description changes #132): an MP4 or MOV H.264 track whose samples switch between avc1 entries at IDR pictures now decodes with each entry's parameter sets in coreav thumbnail, coreav transcode, coreav ffmpeg encodes and tools/decode-oracle; other changes are refused by name (decode.md, "Several sample descriptions").

• Bundle 6 (perf(h264): bundle 6, about 1% faster decoding #106), merged as 2235eadd: band-view lookup and the CAVLC P path, without the edge window. Against its base, eight threads read 0.9888 [0.9814, 0.9913] over 32 training streams and 0.9869 [0.9760, 0.9906] over 64 held-out streams; one thread read 0.9975 over the training streams (CABAC 0.9979, CAVLC 0.9546). Held-out eight-thread time against FFmpeg 9.0.2 is 0.811, faster on 63 of 64 streams. All pre-registered bars are met and output matches on 256 of 256 calls (bundle6/gating). These are round results, not a new official snapshot.

• One band for small pictures at eight threads (docs(h264): record one-band rejection #192), rejected; evidence merged as 7cd77c9d. The valid 32-stream c4 run measured SD discard time at 2.4996 times the baseline (95% interval 1.9522 to 2.7176), missing the registered 0.985 bar, and 2.1582 times FFmpeg 9.0.2, missing parity. Output identity passed 256 of 256 calls. The decoder change was removed; validation and one-thread timing did not run under the committed stop rule. The existing band policy and official snapshot are unchanged (round evidence).

• Opt-in PGO held-out evaluation (docs(pgo): report the held-out evaluation #384), results merged as e2de2ba5: the registered 57-file cohort has class counts 16/16/16/9. Overall one-thread PGO/base is 0.971482 (95% interval 0.965980 to 0.980164), a 2.85% median improvement that misses the fixed 3% bar; all four eight-thread class interval guards pass. The separate CLI diagnostics do not establish their 1% guards, and one-thread FLAC decode is slower in this run. No build or release default changes, no rerun, and no new official snapshot (results and raw calls).

• Conformance acquisition (fix(ci): retain verified conformance downloads #422), merged as 965b737e: CI retains manifest-bound complete and partial archive caches before decoding, verifies SHA-256 on every reuse, and logs bounded network retries. A second digest mismatch fails acquisition. Manifest digests, URL candidates, selected streams and decode bars are unchanged; full fast and safe conformance passed before merge.

Checklist

• Reuse existing residual workspace for interpolation row scratch without changing filtered samples, allocation or scratch accounting (perf(hevc): reuse interpolation row scratch #539).

• Preserve borrowed interpolation row workspace and update private-helper tests when integrating scoped HEVC frame references (perf(hevc): decode ready pictures concurrently #516, perf(hevc): reuse interpolation row scratch #539).

• Preserve scoped neighbour admission and pixel access when integrating HEVC row storage with intra-reference availability reuse (perf(hevc): decode ready pictures concurrently #516, perf(hevc): reuse intra neighbour availability #538).

• Extend checked interior reference-row borrowing to ten-bit interpolation while retaining its shifts, error behavior and memory accounting. (perf(hevc): borrow Main10 reference rows #532).

• Add an interior fractional-luma origin to the automatic/scalar interpolation width-and-phase regression so it directly exercises borrowed Main10 reference rows.

• Measure SIMD predictor-range validation and retain all reference outcomes, including the Main10 timing regression.

• Complete the Main10 predictor-validation follow-up with block-level SIMD dispatch and full automatic and forced-scalar correctness checks.

• Expand combination differential fuzzing to choose the faulty predictor independently of uni/bi selection and offsets.

• Profile ten-bit inverse reconstruction separately before proposing a follow-up change.

• Remove eight-bit interpolation reference-row copies after measuring their narrow instruction cost, keeping filter arithmetic and shared reference-support calculation intact (perf(hevc): borrow interior reference rows #528).

• Profile HEVC neighbour lookup and reuse repeated intra-reference availability checks without changing output (perf(hevc): reuse intra neighbour availability #538).

• Rename the HEVC kernel differential plan file to match its expanded target set without changing campaign membership.

• Explore incremental HEVC row-batch entropy reservations with an exact serial continuation point, without weakening the bin limit.

• Add HEVC tile concurrency after frame scheduling, reusing scheduler machinery only when ownership and filtering stay clear.

• Verify skipped-picture scratch reservations at independent cold admission boundaries, with and without retained worker storage.

• Remove the per-picture full-frame copy when resetting private HEVC wavefront workers while preserving reference and prediction availability.

• Bound the combined HEVC worker count by available CPUs, including explicit requests above host capacity.

• Add bounded HEVC frame-level scheduling without changing decoded output, presentation order, terminal error order or cold memory admission.

• Measure safe HEVC sparse coefficient-level loops against the current decoder while preserving exact bin order and complete output (perf(hevc): skip zero coefficient positions #515).

• Measure the removable HEVC CABAC decision and bypass ceiling, then compare a bounded safe candidate and any assembly candidate with arbitrary-bitstream differential checks. Decision ablation and differential checks are complete; the assembly candidate was closed without merging (perf(hevc): shortcut no-refill CABAC decisions #521). Further CABAC implementation is stopped because no structural candidate has a demonstrated removable ceiling sufficient to proceed; unproved profiling shares are not treated as measured savings.

• Extend coefficient-level differential checks across successive groups with carried contexts and level state.

• Bring the 720p class at one thread to at most 1.00 on the held-out streams.

• Narrow the 1080p one-thread interval so that it lies below 1.00.

• Take a new official snapshot on the same two hosts after each accepted round, and link it here.

• Commit the one-thread bars of the next H.264 round for the fixed-frequency class.

• Calibrate the eight-thread class (c4-standard-16) with turboMode=ALL_CORE_MAX, and adopt it only if it meets pre-registered bars.

• Report the fixed-frequency one-thread class in the next official snapshot.

• Give rounds one shared identity job that fails closed: it checks each decoder's exit status and stops at the first failed decode, instead of hashing the end of a pipe (bundle 3b's identity-job.sh does not check it).

• Make coreav ffprobe -show_frames decode a track with several sample descriptions with each description's parameter sets, as the thumbnail video reader does.

• Let remux, clip and split write every sample entry and keep stsc's description indices, then lift their refusal of tracks with several sample descriptions.

• Make V1's FFmpeg reference command keep the colour description when a stream's colour tags change mid-stream (it converts BT.601 frames to BT.709 there even with -noautoscale), or record that as a known divergence (release crew).

• Measure whether keeping one band at eight threads on small frames improves decoding, with bars committed before the run. Rejected in docs(h264): record one-band rejection #192; the existing band policy is retained.

• Record every stream's coded dimensions and band-rule eligibility in the committed registration before timing.

• Preserve partial VM-job output before a time-limit deletion, and retain completed output independently of the controller, so a controller restart cannot lose the reports.

• Give the H.264 decode example a strict mode that exits nonzero on decode_nal or flush errors, and use it in the shared round identity job.

• Measure the musl release binary's decode time with and without mimalloc on GitHub Actions as a reference figure, if a public number is wanted.

Not in scope

• aarch64 (NEON): H.264 decode speed on aarch64 (NEON) #23.
• Decoding codecs other than H.264, and GPU decoding.