Skip to content
Merged
5 changes: 4 additions & 1 deletion doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -766,7 +766,10 @@ encoded, and a buffered player its whole lookahead, while native playback
decodes everything at once. The browser's decode gate releases a playing ring's
next frame while what it was given, landed or still decoding, reaches less than
its floor and one window past the playhead, so the ring always holds a full
window to stretch, and a hole in the media plays as silence.
window to stretch, and a hole in the media plays as silence. A frame stamped
within a millisecond and two samples of where the last one ended continues it
in both players: millisecond stamps, or ones rounded from another clock, land
either side of the exact boundary, which is no hole.

A zero `stretch` turns the stretch off, for content where a pitch-period splice
is audible, such as music. Every window holds, so received audio plays
Expand Down
11 changes: 7 additions & 4 deletions js/watch/src/audio/replay.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -159,8 +159,11 @@ async function lag(trace: Arrival[], options: Options, settle: number) {
return { lags, median: lags[lags.length >> 1], underruns, trimmed, accelerated };
}

/** The shortest pitch period a seam moves by: 2.5 ms. A lag shorter than that cannot be corrected. */
const PERIOD = 2.5;
/**
* The longest pitch period of the replay's speech, at its 90 Hz floor: a seam moves by one period, so
* a lag shorter than that may rest uncorrected.
*/
const PERIOD = 1000 / 90;

// Audio and video keep separate clocks that align once the delay covers the arrivals. The ring plays
// onto the shared clock: in steady state the playhead rests within a pitch period of it, either way.
Expand Down Expand Up @@ -373,8 +376,8 @@ describe("buffered replay against the shared clock", () => {
lookahead: 500,
duration: observed(30),
};
// NetEq's accelerate takes one pitch period per stretch, so this 2.5 ms tone gains about 30 ms a
// second: the stall's lag is gone well before 20 s.
// NetEq's accelerate takes one pitch period per stretch, at least 4.5 ms of this speech, so it
// gains well over 100 ms a second: the stall's lag is gone well before 20 s.
const result = await play(trace, options, (at) => (at >= 2000 && at < 5000) || at >= 20_000);
expect(result.min).toBeGreaterThanOrEqual(-PERIOD);
expect(result.max).toBeLessThanOrEqual(PERIOD);
Expand Down
70 changes: 68 additions & 2 deletions js/watch/src/audio/replay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,70 @@ export interface Quantum {
accelerated: number;
}

/**
* A seeded speech-like signal, a pure function of media time so every replay hears the same audio.
*
* A constant gives the stretch's pitch search its shortest period everywhere, so catching up removes
* one 2.5 ms period a window; speech's longer, varying periods and pauses let it catch up as fast as
* a real talker does. Syllables of 120 to 300 ms glide the fundamental between 90 and 220 Hz under a
* raised-cosine level, with pauses of 100 to 400 ms between some. The floor stays above the harness's
* -60 dBFS silence classifier, and no sample is exactly zero, which the output tap reads as missing.
*/
export class Speech {
readonly #rate: number;
readonly #origin: number;
#samples = new Float32Array(0);
#length = 0;
#phase = 0;
#seed = 0x5eed;

/** `origin` is the earliest media timestamp read, in ms. */
constructor(rate: number, origin: number) {
this.#rate = rate;
this.#origin = origin;
}

/** `count` samples starting at media timestamp `timestamp`, in ms. */
read(timestamp: number, count: number): Float32Array {
const start = Math.round(((timestamp - this.#origin) * this.#rate) / 1000);
if (start < 0) throw new Error(`speech read at ${timestamp} ms, before its ${this.#origin} ms origin`);
while (this.#length < start + count) this.#syllable();
return this.#samples.slice(start, start + count);
}

/** Mulberry32: uniform in [lo, hi). */
#random(lo: number, hi: number): number {
this.#seed = (this.#seed + 0x6d2b79f5) | 0;
let t = this.#seed;
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return lo + (((t ^ (t >>> 14)) >>> 0) / 4294967296) * (hi - lo);
}

#syllable() {
const FLOOR = 0.03;
const pause = this.#random(0, 1) < 0.25;
const count = Math.round((this.#rate * (pause ? this.#random(100, 400) : this.#random(120, 300))) / 1000);
const from = this.#random(90, 220);
const to = this.#random(90, 220);
const peak = pause ? FLOOR : this.#random(0.1, 0.5);
if (this.#samples.length < this.#length + count) {
const grown = new Float32Array(Math.max(2 * this.#samples.length, this.#length + count));
grown.set(this.#samples.subarray(0, this.#length));
this.#samples = grown;
}
for (let i = 0; i < count; i++) {
const x = i / count;
this.#phase += (2 * Math.PI * (from + (to - from) * x)) / this.#rate;
const level = FLOOR + (peak - FLOOR) * 0.5 * (1 - Math.cos(2 * Math.PI * x));
const p = this.#phase;
const value = level * (Math.sin(p) + 0.5 * Math.sin(2 * p) + 0.25 * Math.sin(3 * p)) * 0.6;
this.#samples[this.#length + i] = Math.abs(value) < 1e-6 ? 1e-6 : value;
}
this.#length += count;
}
}

/** Samples as the milliseconds `AudioBuffer` posts to the worklet. */
const millis = (rate: number, samples: number) => (samples / rate) * 1000;

Expand All @@ -117,7 +181,7 @@ const settle = () => new Promise<void>((resolve) => setImmediate(resolve));
* Play `trace` in simulated real time, yielding every render quantum from the first arrival to the
* end of the observation.
*
* A group's stream finishes with its last recorded frame. The audio is a constant, never zero, so
* A group's stream finishes with its last recorded frame. The audio is {@link Speech}, never zero, so
* what the ring did not supply is exactly the zeros at the end of a quantum, which is how the
* harness's output tap reads a real page.
*/
Expand All @@ -139,6 +203,8 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato

// Where each group's stream finishes.
const ends = new Map<number, number>();
const origin = trace.reduce((min, a) => Math.min(min, a.timestamp), Infinity);
const speech = new Speech(rate, origin);
trace.forEach((arrival, i) => {
ends.set(arrival.group, i);
});
Expand Down Expand Up @@ -239,7 +305,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
ring.received(timestamp + micros(entry.duration));
await gate.wait(timestamp, abort.signal);
if (abort.signal.aborted) return;
const pcm = new Float32Array(Math.round((entry.duration * rate) / 1000)).fill(0.5);
const pcm = speech.read(entry.timestamp, Math.round((entry.duration * rate) / 1000));
ring.write(timestamp, [pcm]);
// The decoder reports each output to the gate the way `AudioBuffer` and `Decoder.#emit` do.
const end = timestamp + micros((pcm.length / rate) * 1000);
Expand Down
27 changes: 27 additions & 0 deletions js/watch/src/audio/ring-buffer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,22 @@ describe("writing data", () => {
}
});

it("plays frames stamped within a millisecond of contiguous as contiguous", () => {
const buffer = exact({ rate: 48000, channels: 1, latency: 10 });
// 480 samples is 10 ms. The second frame is stamped 1 ms late, the third 2 samples early (of
// the second's actual end), and the fourth after a real 2 ms hole.
write(buffer, 0, 480, { value: 1 });
write(buffer, 11, 480, { value: 2 });
write(buffer, (960 - 2) / 48, 480, { value: 3 });
write(buffer, (1440 + 96) / 48, 480, { value: 4 });
expect(buffer.length).toBe(4 * 480 + 96);

const output = read(buffer, 4 * 480 + 96, 1)[0];
expect(output.subarray(0, 1440).every((v, i) => v === 1 + Math.floor(i / 480))).toBe(true);
expect(output.subarray(1440, 1536).every((v) => v === 0)).toBe(true);
expect(output.subarray(1536).every((v) => v === 4)).toBe(true);
});

// These drain the ring below its floor without refilling it, which the stretch slows down for, so
// they run exact.
it("should handle late-arriving samples (out-of-order writes)", () => {
Expand Down Expand Up @@ -555,6 +571,17 @@ describe("declared end", () => {
expect(buffer.silent).toBe(false);
});

it("ends where contiguous frames end, not at their jittered stamp", () => {
const buffer = new AudioRingBuffer({ rate: 48000, channels: 1, latency: 10 });
// The second frame is stamped 1 ms late, so it plays from sample 480 and ends at 960, not 1008.
buffer.end(21_000);
write(buffer, 0, 480, { channels: 1 });
write(buffer, 11, 480, { channels: 1 });
expect(read(buffer, 1024, 1)[0].length).toBe(960);
expect(buffer.silent).toBe(true);
expect(buffer.underruns).toBe(0);
});

it("counts running dry short of the end as an underrun", () => {
const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 20 });
write(buffer, 0, 20, { channels: 1 });
Expand Down
23 changes: 19 additions & 4 deletions js/watch/src/audio/ring-buffer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,11 @@ export class AudioRingBuffer {
this.#writeIndex = start;
this.#anchored = true;
this.#level = Math.max(this.#received ?? start, start) - start;
} else if (this.#contiguous(start)) {
// A frame starting within a millisecond and two samples of where the last one ended
// continues it: timestamps stamped in milliseconds, or rounded from another clock, land on
// either side of the exact boundary, and a hole there would play a few samples of silence.
start = this.#writeIndex;
}

// Ignore samples behind the read index, which runs ahead of the playhead by the staged window:
Expand Down Expand Up @@ -294,10 +299,6 @@ export class AudioRingBuffer {
// Fill gaps with zeros if there's a discontinuity
if (start > this.#writeIndex) {
const gapSize = Math.min(start - this.#writeIndex, this.#buffer[0].length);
if (gapSize === 1) {
console.warn("floating point inaccuracy detected");
}

for (let channel = 0; channel < this.channels; channel++) {
const dst = this.#buffer[channel];
for (let i = 0; i < gapSize; i++) {
Expand Down Expand Up @@ -325,6 +326,7 @@ export class AudioRingBuffer {
if (end > this.#writeIndex) {
this.#writeIndex = end;
}
this.#snapEnd();
// Decoded audio arrived too, if no message said so first.
this.received((end / this.rate) * 1_000_000);
this.#resume();
Expand Down Expand Up @@ -371,11 +373,24 @@ export class AudioRingBuffer {
*/
end(timestamp: number): void {
this.#end = Math.round((timestamp / 1_000_000) * this.rate);
this.#snapEnd();
// The end can arrive after the playhead already ran out at it, or before any audio, as for a
// listener that joins during a mute.
if (!this.#anchored || (this.#stalled && this.length <= 0 && this.#ended())) this.#silent = true;
}

// Whether `index` is within a millisecond and two samples of the write index: the jitter of
// timestamps stamped in milliseconds or rounded from another clock.
#contiguous(index: number): boolean {
return Math.abs(index - this.#writeIndex) <= this.rate / 1000 + 2;
}

// Move a declared end within that jitter of the written audio onto its end, since frames are placed
// contiguously rather than at their stamps.
#snapEnd(): void {
if (this.#anchored && this.#end !== undefined && this.#contiguous(this.#end)) this.#end = this.#writeIndex;
}

// Move the read index forward to `index`, dropping any queued audio it passes, staged included,
// and the front of the incoming chunk at `start` when `index` lies past it. The caller counts the
// skip; an empty ring only re-anchors over a gap it never held, so the trim does not move, as in
Expand Down
30 changes: 0 additions & 30 deletions quest/m1/audio-jitter-target/speech-replay.md

This file was deleted.

15 changes: 14 additions & 1 deletion rs/moq-cli/src/play/timeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -130,13 +130,20 @@ impl AudioTimeline {
// keeps the next frame contiguous with the new timeline position.
let origin = *self.origin.get_or_insert(start);
let expected = (start.saturating_sub(origin).as_secs_f64() * sample_rate as f64).round() as u64;
// A hole within that rounding is the same jitter, not missing audio: the frame continues
// the last one rather than playing a few samples of silence ahead of it.
let hole = expected.saturating_sub(self.written);
let hole = if (hole as f64) <= tolerance.as_secs_f64() * sample_rate as f64 {
0
} else {
hole
};
let skipped = hole > fill_max;
let silence = if skipped { 0 } else { hole };
let reset_sink = rewound || skipped;
self.written = self
.written
.max(expected)
.saturating_add(hole)
.saturating_add(u64::try_from(samples).unwrap_or(u64::MAX));
self.end = Some(end);

Expand Down Expand Up @@ -234,6 +241,12 @@ mod tests {
// 1024 frames end at 23.22 ms, but an FLV timestamp carries 23 ms.
let rounded = timeline.push(Duration::from_millis(23), 1024, 44_100, 22_050);
assert!(!rounded.reset_sink);
assert_eq!(rounded.silence, 0);

// The third frame starts at 46.44 ms, a 47 ms stamp: no silence ahead of it either.
let late = timeline.push(Duration::from_millis(47), 1024, 44_100, 22_050);
assert!(!late.reset_sink);
assert_eq!(late.silence, 0);
}

#[test]
Expand Down
9 changes: 5 additions & 4 deletions test/audio-quality/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,10 +234,11 @@ writes the same samples a page does, so `analyze.ts` reduces both alike, and the
and clock checks have nothing to void. Decoding is taken as instant. A replay is deterministic, so
its budgets are exactly what it measured.

The ring time-stretches live and buffered playback as the page's does, but the replay's audio is a constant, so
every seam admits at the shortest pitch period, 2.5 ms. Accelerate therefore moves less per stretch
than it would from speech; only fast accelerate, which takes every whole period that fits, moves its
most.
The ring time-stretches live and buffered playback as the page's does, on a seeded speech-like signal
rather than the decoded audio: syllables of 120 to 300 ms glide a 90 to 220 Hz fundamental under a
varying level, with pauses between some held above the silence classifier and no sample exactly
zero. A seam therefore removes one speech-length pitch period, so a replay's catch-up after a stall
moves at the pace speech would, not a constant tone's 2.5 ms a window.

```bash
just test audio-quality-record # the three recorded lanes, 35 s each, needs the network
Expand Down
Loading