Skip to content

docs(quest): accept the log lock profile and delete its quest - #7742

Merged
kixelated merged 8 commits into
mainfrom
quest/m1/unbounded-groups/log-lock-profile
Nov 3, 2026
Merged

docs(quest): accept the log lock profile and delete its quest#7742
kixelated merged 8 commits into
mainfrom
quest/m1/unbounded-groups/log-lock-profile

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Problem

The log-lock quest needed evidence that the epoch-based kio::log change in moq-dev/moq#6782 causes no lock-contention regression. The first attempt on desktop was mixed: the GroupState share dropped, but absolute wait rose at 16 workers (47 -> 599 us). A quiet-window repeat on nixos2 found no quiet window. Its one completed pre-epoch run shows the contrary wait on the same stack before #6782.

Conclusion (accepted by the maintainer, 2026-11-03): no lock-contention regression attributable to moq-dev/moq#6782. The pre-epoch run on nixos2 shows the same GroupState wait stack (Consumer::poll_read_frame_once -> poll_read_frame -> poll<GroupState>, 709 µs) as the epoch run on desktop (599 µs). The figures come from different hosts and are not comparable; no quiet same-host repeat was completed. This PR deletes the quest with quest rm, which removes its link from the unbounded-groups epic. It also records the conclusion in that epic's storage decision.

Approach (desktop attempt)

Ran the unchanged current just bench-profile lock video-heavy "1 4 16" tokio-workers <base> recipe twice through quest gate, using the pre-epoch parent and the epoch merge. Both used the current generator and harness at the claim commit, whose tree equals its main parent. The quest now records the evidence, refusal, and remaining maintainer choice.

desktop: 24 logical CPUs, perf_event_paranoid=-1, initial one-minute load 0.21, no other running T3 agents in two all-project checks. The recipe pins to performance CPUs 0-7, so 16 workers oversubscribe that allowed set as the documented recipe specifies. No continuous independent host-idle trace was collected, so this is a quiet-start, no-other-agent attempt, not proof that every background activity stayed absent. All builds and captures ran sequentially; no reviewer ran during profiling.

Nix dev shell, rustc 1.98.1, profiling profile (release, fat LTO, one codegen unit, line-table debug info, no stripping), -C force-frame-pointers=yes, system allocator, default features plus moq-relay/io-uring,moq-bench/uring; the selected runtime was verified as Tokio workers, with the requested exact count. All source dirty patches were empty and untracked-source lists empty.

The workload remains 256 connections, one broadcast each, two subscriptions, 60 fps, 1200-byte frames, group_size=59. Each control precedes its capture; capture duration is 5 s. Valid profile summaries report zero lifetime loss. This is one observation per historical revision/worker count, not a statistical estimate of a small difference.

Results on desktop (first attempt)

Durable reduced evidence: part 1, part 2, part 3, part 4, part 5, part 6, part 7. These comments retain the full folded stacks, capture/runtime summaries, tool and source context, failure diagnostic, and build-tool incident log.

A = run.AW5tm7, pre-epoch base; B = run.nLyO6T, epoch base. current has the same source in both runs, rebuilt through different Cargo entrypoints as disclosed below. Waits exclude spinning. The denominator is all worker std-mutex blocked time, not CPU time.

Run / revision workers control fps profiled fps total mutex ms GroupState us GroupState share
A / pre-epoch 1 30717.8 31488.0 0.000 0 undefined (zero total)
A / pre-epoch 4 30714.3 31481.5 1.590 8 0.50%
A / pre-epoch 16 31339.8 31512.3 0.401 47 11.72%
B / epoch 1 31489.9 30719.8 0.000 0 undefined (zero total)
B / epoch 4 31682.6 30726.1 7.612 0 0.00%
B / epoch 16 30725.7 31440.8 33.722 599 1.78%
A / current 1 31284.4 31251.3 0.000 0 undefined (zero total)
A / current 4 31135.0 30710.1 2.293 26 1.13%
A / current 16 31149.2 30723.8 61.226 937 1.53%
B / current 1 30717.8 31128.0 0.000 0 undefined (zero total)
B / current 4 30709.4 30720.5 3.260 18 0.55%
B / current 16 retained in evidence refused unavailable unavailable unavailable

Folded stacks truncate each interval to microseconds; the total mutex denominator retains nanoseconds from lock.json. Attribute by the innermost lock<T> frame: kio::channel::Waiters is separate even if its callers mention GroupState. Full folds and lock/runtime/control summaries are retained in the evidence comments below, so scratch cleanup does not erase the reduced evidence. The multi-gigabyte perf.data files remain local scratch, not durable attachments.

Historical wait-by-stack diff

Every nonzero GroupState acquisition is below. Common leaf suffix: lock<GroupState> (or lock<kio::State<GroupState>>) -> Mutex::lock_contended -> futex_wait -> syscall. Full generic and scheduler frames are preserved in the evidence comments.

workers / acquisition stack before us epoch us delta us
1 / all GroupState stacks 0 0 0
4 / GroupServe::poll_serve -> Handle::cache_refresh -> read<GroupState> 5 0 -5
4 / Consumer::poll_read_frame_once -> Cursor::poll_read_frame -> poll<GroupState> 3 0 -3
16 / GroupServe::poll_serve -> Handle::cache_refresh -> read<GroupState> 47 0 -47
16 / Consumer::poll_read_frame_once -> Cursor::poll_read_frame -> poll<GroupState> 0 599 +599

At 4 workers the group-state wait falls to none. At 16 workers its share drops 11.72% -> 1.78%, while its absolute wait rises 47 -> 599 us. The denominator grew 0.401 -> 33.722 ms; most folded wait is TrackState (347 -> 32128 us). Calling this an unqualified no-contention-regression result from share alone would hide the contrary absolute measurement. It also does not refute the earlier Criterion per-reader cost numbers: blocked time under this live workload measures something different.

The post-epoch source still takes the state lock when the log cannot serve a frame, including its live edge: Cursor::poll_read_frame. A blocked stack alone does not identify which branch or a root cause, and one capture cannot establish that the epoch change caused the increase. Recommendation at the time (since decided below): keep the quest open and authorize alternating historical captures with warm direct-Cargo builds, recording host load per capture and verifying no other agents run throughout, preserving the existing workload, worker matrix, and validity checks. The 16-worker contrast is the unresolved signal. Make no production change from these samples.

Current-source observations and refusal

At 4 workers both current captures still have group-state waits in poll_read_frame_once<..., Prefetch::pop_cached> -> poll<GroupState> (26 and 18 us). At 16 workers the first current capture has 937 us in total group-state waits; the full stack evidence is below. These candidate observations identify the Plan's possible epoch-versus-main investigation, but that suggestion remains conditional on reproducing the waits with established per-capture quietness; they do not establish a later regression. The 16-worker second-current capture was refused:

Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.
profile: worker moq-quic-0 (1448077) has nonalternating switch records

It has no lock.json, no folded-stack result, and no accepted profiled throughput. The command exited 1; it was not retried or relaxed. The first command exited 0. Successful captures passed the existing switch-window, symbol-resolution, and dropped-event validation; no invalid capture enters the tables.

Repeat on nixos2 (2026-11-03)

Evidence: part 1 (reduced rows with per-capture host fields, and the full pre-epoch 16-worker fold) and part 2 (4-worker fold, void runs, run log).

Host. desktop was running many agents (load 25 to 31) and nixos ran the land worker and CI, so the repeat used nixos2: i7-8700, 6 cores / 12 threads, no hybrid cores (so no pinning; affinity 0-11), perf_event_paranoid=-1, kernel 6.18. That replaces the quest's decided desktop host for this repeat, per the maintainer's newest brief. Host configuration was not changed. Its ~/.cargo/config.toml jobs = 2 was left in force. The scratch checkout lived under ~/moq-7742-profile at the claim commit b5a2e8fb77, the same harness and current source as the desktop attempt. It was removed afterwards, and nothing of this run is left running.

Builds. Direct Nix Cargo/rustc 1.98.1 and perf 7.3.0 from the dev shell, with no shim (type -a cargo resolved to the Nix store) and BASH_ENV=/dev/null. Warm-up built current and both bases into the recipe's own target directories with the recipe's flags, before any capture. Each run still rebuilds its base's workspace crates (about 5 minutes), because the harness checks the base out at a fresh path. Within a run, all builds finish before the first capture.

Quietness. Each run started only after no agent process had a running tool command for one minute (uptime load 0.10 to 0.18 at each start). A 2 s monitor recorded load, whole-host busy cores, CPU used outside the run's process tree, other agents' tool commands, and foreign build processes. Each capture's window runs from its control's first file to its profiled run's last file.

Run Revision Outcome Foreign activity during captures
A1 pre-epoch d854bf98b1~1 void; base 16 refused (nonalternating switch records) a new agent session's cargo/rustc builds, 0.2 to 1.0 cores mean
r1A pre-epoch d854bf98b1~1 base 1/4/16 and current 1/4 captured; current 16 refused (nonalternating) other agents' light tool commands (codex, git, gh, quest, bash); 0.06 to 0.21 cores mean, 1.03 peak; no foreign builds
r2B epoch d854bf98b1 void; base 16 refused (perf dropped 1 event) another session's moq bench builds (relay-cpu-bisect), 5.1 to 5.8 cores mean throughout

Only r1A has usable historical captures, and even it fails the "no other agents running throughout" bar; its foreign CPU was low. The run log, pre and post uptime, and every row are in the evidence comments. The alternating sequence stopped after r2B: three other agent sessions and a detached CPU-pinned bisect benchmark were active on nixos2, so further runs would neither be quiet nor leave that benchmark undisturbed.

r1A capture workers control fps profiled fps loss total mutex ms GroupState us GroupState share load1 max foreign cores mean / max
pre-epoch 1 22069.8 23220.6 37% 0.000 0 undefined 3.67 0.07 / 0.14
pre-epoch 4 31483.3 30681.2 0 0.822 9 1.10% 5.21 0.06 / 0.16
pre-epoch 16 30295.3 30940.0 0 38.819 709 1.83% 12.77 0.21 / 1.03
current 1 20632.9 21438.0 33% 0.000 0 undefined 11.83 0.03 / 0.10
current 4 31239.6 31665.4 0 3.489 0 0.00% 9.07 0.05 / 0.71
current 16 31216.5 refused 7.99 0.05 / 0.27

Load includes the run's own relay and generator. At 1 worker this CPU delivers only 20 to 23k of the 31k offered frames per second, so 1-worker captures on nixos2 fail the zero-loss check (they have no mutex wait either way). The pre-epoch 16-worker wait is entirely Consumer::poll_read_frame_once -> poll_read_frame -> poll<GroupState>. That is the stack the desktop epoch capture showed at 599 us, where the desktop pre-epoch capture had none on it. The pre-epoch 4-worker wait is Consumer::poll_expiry -> read<GroupState>. Total mutex wait is again mostly TrackState (37906 of 38819 us at 16 workers). The current 4-worker capture had no GroupState wait, unlike both desktop current 4-worker captures (26 and 18 us under Prefetch::pop_cached).

Conclusion: still inconclusive, but the contrary signal is weaker. The 47 -> 599 us desktop contrast is not attributable to #6782: the same stack waits 709 us on the pre-epoch revision here. Different hosts' totals are not comparable (0.401 ms before on desktop, 38.8 ms here), so no same-host before/after pair exists yet. The share goal ("drops or holds") therefore has no valid measurement. Repeated 16-worker refusals (nonalternating switch records twice, a dropped event once) also make 16-worker pairs expensive to complete.

Impact

  • Production, public API and wire: none. No version bump or migration.
  • Quest only: deletes quest/m1/unbounded-groups/log-lock-profile.md and its epic link, and records the accepted conclusion in quest/m1/unbounded-groups/README.md. Main was merged into the branch first (no rebase).

Alternatives and choices

  • Rejected calling the lower share sufficient proof when absolute group-state wait rose.
  • Rejected changing production code, weakening capture validation, or dispatching Nightly.
  • nixos2 repeat: ran the unchanged recipe with TMPDIR set to a scratch directory beside (not inside) the scratch checkout, so the harness's base worktree inherited no checkout Cargo configuration and needed no bwrap. Stopped the sequence rather than run unquiet captures, and did not touch the other sessions on the host.
  • Desktop attempt: kept all scratch physically under this checkout's .tmp/. The first attempt failed before building because a nested historical checkout inherited the root Cargo alias with an incompatible type. The successful commands used bwrap --bind / / --dev-bind /dev /dev --bind "$PWD/.tmp/profile-tmp" /tmp, so the recipe's temporary checkout stays in owned scratch but Cargo sees no parent checkout configuration. No source or harness patch was applied.
  • The canonical local quest branch remains checked out in an older worktree. This checkout retained its T3 branch, tracked the new upstream claim, and pushes an explicit refspec to the claimed PR branch. The older worktree was untouched.

Build-tool incident (desktop attempt)

The initial Nix-shell cargo entrypoint was an installed mbx shim. Its automatic capacity GC ran during builds before the first workload, contrary to the task's no-shared-GC constraint. The log reports 7 managed target directories removed (7.4 GiB logical; 6 abandoned, 1 live), 5.4 GiB of other cache objects evicted, and about 30.6 GiB of explicitly shared cache objects evicted. I did not invoke GC or delete a checkout, but failed to inspect that entrypoint before building. This cannot be undone; affected build artifacts can be regenerated. The second recipe and local CI bypassed it using the pinned Nix Cargo/rustc directory first in PATH and BASH_ENV=/dev/null. No cache-manager output appeared in the second recipe. Initial free disk was 12.2 GB; no actual storage failure occurred.

Checks

  • quest check --base origin/main with the flake-pinned quest (nix shell .#quest): passed, 580 documents. The globally installed quest 0.1.0 rejects the Waiting heading in 19 other quests already on main, which is pre-existing tool-version drift. That is also why quest rm exited 1 after it had deleted the file and fixed the link.
  • quest gate nix develop --command just check origin/main: passed (biome, markdown, gh check and test, quest, shell, large files, text, toml, nix, justfile).
  • Evidence runs: both desktop historical 1/4/16-worker captures passed validation; on nixos2, only r1A's pre-epoch 1/4/16 and current 1/4 captures did (1 worker fails zero-loss on that CPU).

Review

Skipped for head 686ba58acb: the change since 37d59f306f is a one-line doc fix to the epic README. It names the host beside each wait figure and says they are not comparable, per Pullfrog's thread. 37d59f306f was itself a docs-only head with the review skipped. The last review round was on d59c481488 (Claude Code, Claude Opus 5.5, medium effort, quest review --base origin/main, round 3), and it reported no findings.

  • Pullfrog (quest/m1/unbounded-groups/README.md:120, cross-host 709 vs 599 µs read as a controlled comparison): fixed in 686ba58acb, mirrored in the Problem section above.

Decisions

  • 2026-11-03, maintainer: resolve the contrary 16-worker absolute wait before calling this a no-regression result. ✅ Repeat the two historical revisions in alternating order on a quiet window; accept the share-only goal; skip. Done on nixos2, which had no quiet window (see above).
  • 2026-11-03, maintainer: how to finish the comparison. Reserve a quiet window; ✅ accept "no regression attributable to perf(kio,net): lock-free append-only log for group frames #6782" and delete the quest; drop without a conclusion. They offered kixtop as a quiet host, but it is a macOS laptop without Linux perf, so it cannot run this profile; accepting was the stated alternative.

Follow-ups

  • Exclusive benchmark-host reservation: this repeat and a detached relay-cpu-bisect instruction benchmark started on nixos2 in the same minute (19:13Z). r2B's captures (19:19 to 19:23Z) overlapped that benchmark's build phase. That session should check whether its own captures overlapped r2B. A host lock that quest gate or the bench recipes take would prevent this.
  • The 16-worker nonalternating switch-record and lost-record refusals reproduced on nixos2 too (r1A current, A1 base under load, r2B base). Cluster route memory already plans that harness fix, so no new quest is needed.
  • No epoch-versus-main quest: the r1A current 4-worker capture had no cached-read GroupState wait, so the desktop candidate did not persist.
  • The flake-pinned quest accepts ## Waiting, but the installed quest 0.1.0 does not; a quest pin bump will need to migrate those 19 sections.

(Written by Claude Opus 5.5)

kixelated and others added 4 commits November 3, 2026 05:15
@kixelated
chore(quest): claim quest/m1/unbounded-groups/log-lock-profile.md
b5a2e8f 
Quest-Claim: 105be10455aa5d77
@kixelated @codex
docs(quest): record mixed quiet-host log lock evidence
b723d92 
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated @codex
docs(quest): qualify profiling evidence and pin current source
a648cda 
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated @codex
docs(quest): make the conditional follow-up durable
d59c481 
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 1/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.AW5tm7, base, 1 workers

{
  "source_revision": "7237beb17a9405642828d954ea743d5b663172a8",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5099026692,
    "workers": {
      "1439025": {
        "comm": "moq-quic-0",
        "switches": 5948,
        "cpu_ns": 3870514450,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 1205015967,
        "mutex_wait_per_cpu": 0.0
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15358.6,
      "recv_fps": 30717.8,
      "send_mbps": 147.44256,
      "recv_mbps": 294.89088,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 0.763365269461078,
      "cpu_us_per_frame": 24.850909552802545,
      "throughput_window_start_ms": 1793712361199,
      "throughput_window_end_ms": 1793712366199,
      "host_window_start_ms": 1793712361199,
      "host_window_end_ms": 1793712366199,
      "threads": 4,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15360,
      "recv_fps": 31488,
      "send_mbps": 147.456,
      "recv_mbps": 302.2848,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 0.7664151696606785,
      "cpu_us_per_frame": 24.33991265436606,
      "throughput_window_start_ms": 1793712371275,
      "throughput_window_end_ms": 1793712376275,
      "host_window_start_ms": 1793712371275,
      "host_window_end_ms": 1793712376275,
      "threads": 4,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 0,
    "group_share_pct": null,
    "wait_by_type_us": {},
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

(empty: no mutex waits recorded)

run.AW5tm7, base, 16 workers

{
  "source_revision": "7237beb17a9405642828d954ea743d5b663172a8",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-10",
      "moq-quic-11",
      "moq-quic-12",
      "moq-quic-13",
      "moq-quic-14",
      "moq-quic-15",
      "moq-quic-2",
      "moq-quic-3",
      "moq-quic-4",
      "moq-quic-5",
      "moq-quic-6",
      "moq-quic-7",
      "moq-quic-8",
      "moq-quic-9"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5100340036,
    "workers": {
      "1439451": {
        "comm": "moq-quic-0",
        "switches": 13598,
        "cpu_ns": 477916443,
        "mutex_wait_ns": 279490,
        "other_futex_wait_ns": 3491,
        "other_off_cpu_ns": 4534460652,
        "mutex_wait_per_cpu": 0.0005848093408244588
      },
      "1439452": {
        "comm": "moq-quic-1",
        "switches": 11842,
        "cpu_ns": 391567862,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4620172966,
        "mutex_wait_per_cpu": 0.0
      },
      "1439453": {
        "comm": "moq-quic-2",
        "switches": 11904,
        "cpu_ns": 340960779,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 60950,
        "other_off_cpu_ns": 4669967674,
        "mutex_wait_per_cpu": 0.0
      },
      "1439454": {
        "comm": "moq-quic-3",
        "switches": 8660,
        "cpu_ns": 237012313,
        "mutex_wait_ns": 7131,
        "other_futex_wait_ns": 202766,
        "other_off_cpu_ns": 4771557941,
        "mutex_wait_per_cpu": 3.008704446506963e-05
      },
      "1439455": {
        "comm": "moq-quic-4",
        "switches": 13103,
        "cpu_ns": 400334485,
        "mutex_wait_ns": 10630,
        "other_futex_wait_ns": 546095,
        "other_off_cpu_ns": 4610797783,
        "mutex_wait_per_cpu": 2.6552796219890973e-05
      },
      "1439456": {
        "comm": "moq-quic-5",
        "switches": 13463,
        "cpu_ns": 448533245,
        "mutex_wait_ns": 16993,
        "other_futex_wait_ns": 455848,
        "other_off_cpu_ns": 4563890194,
        "mutex_wait_per_cpu": 3.788570900692099e-05
      },
      "1439457": {
        "comm": "moq-quic-6",
        "switches": 12564,
        "cpu_ns": 371581826,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 92893,
        "other_off_cpu_ns": 4639891849,
        "mutex_wait_per_cpu": 0.0
      },
      "1439458": {
        "comm": "moq-quic-7",
        "switches": 12449,
        "cpu_ns": 388060020,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 66625,
        "other_off_cpu_ns": 4624758687,
        "mutex_wait_per_cpu": 0.0
      },
      "1439459": {
        "comm": "moq-quic-8",
        "switches": 12518,
        "cpu_ns": 479194839,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 486044,
        "other_off_cpu_ns": 4534741605,
        "mutex_wait_per_cpu": 0.0
      },
      "1439460": {
        "comm": "moq-quic-9",
        "switches": 12736,
        "cpu_ns": 411446384,
        "mutex_wait_ns": 13551,
        "other_futex_wait_ns": 21307,
        "other_off_cpu_ns": 4600457122,
        "mutex_wait_per_cpu": 3.293503242940154e-05
      },
      "1439461": {
        "comm": "moq-quic-10",
        "switches": 12045,
        "cpu_ns": 386481105,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4625104278,
        "mutex_wait_per_cpu": 0.0
      },
      "1439462": {
        "comm": "moq-quic-11",
        "switches": 13160,
        "cpu_ns": 413069803,
        "mutex_wait_ns": 22490,
        "other_futex_wait_ns": 15773,
        "other_off_cpu_ns": 4598957296,
        "mutex_wait_per_cpu": 5.4446003645538813e-05
      },
      "1439463": {
        "comm": "moq-quic-12",
        "switches": 13419,
        "cpu_ns": 421463988,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 25154,
        "other_off_cpu_ns": 4590727782,
        "mutex_wait_per_cpu": 0.0
      },
      "1439464": {
        "comm": "moq-quic-13",
        "switches": 12895,
        "cpu_ns": 439529292,
        "mutex_wait_ns": 47141,
        "other_futex_wait_ns": 9220,
        "other_off_cpu_ns": 4572504922,
        "mutex_wait_per_cpu": 0.00010725337504923335
      },
      "1439465": {
        "comm": "moq-quic-14",
        "switches": 13222,
        "cpu_ns": 408671008,
        "mutex_wait_ns": 3659,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4603439711,
        "mutex_wait_per_cpu": 8.95341222737288e-06
      },
      "1439466": {
        "comm": "moq-quic-15",
        "switches": 12696,
        "cpu_ns": 367026764,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 29290,
        "other_off_cpu_ns": 4643900062,
        "mutex_wait_per_cpu": 0.0
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15359.8,
      "recv_fps": 31339.8,
      "send_mbps": 147.45408,
      "recv_mbps": 300.86208,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 1.2505290136857758,
      "cpu_us_per_frame": 39.902265288412046,
      "throughput_window_start_ms": 1793712417304,
      "throughput_window_end_ms": 1793712422304,
      "host_window_start_ms": 1793712417304,
      "host_window_end_ms": 1793712422304,
      "threads": 19,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15363.072614522905,
      "recv_fps": 31512.3024604921,
      "send_mbps": 147.4854970994199,
      "recv_mbps": 302.51810362072416,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 1.347653933296381,
      "cpu_us_per_frame": 42.76596211863523,
      "throughput_window_start_ms": 1793712427882,
      "throughput_window_end_ms": 1793712432881,
      "host_window_start_ms": 1793712427882,
      "host_window_end_ms": 1793712432881,
      "threads": 19,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 47,
    "group_share_pct": 11.718214343593004,
    "wait_by_type_us": {
      "TrackState": 347,
      "GroupState": 47
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 279
[unknown];<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Handle>::cache_refresh;read<moq_net::model::group::GroupState>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 47
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 18
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 9
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 2/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.AW5tm7, base, 4 workers

{
  "source_revision": "7237beb17a9405642828d954ea743d5b663172a8",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-2",
      "moq-quic-3"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5100032948,
    "workers": {
      "1439230": {
        "comm": "moq-quic-0",
        "switches": 13636,
        "cpu_ns": 1315505604,
        "mutex_wait_ns": 486336,
        "other_futex_wait_ns": 2704741,
        "other_off_cpu_ns": 3711019040,
        "mutex_wait_per_cpu": 0.00036969511837974656
      },
      "1439231": {
        "comm": "moq-quic-1",
        "switches": 14442,
        "cpu_ns": 1210719532,
        "mutex_wait_ns": 242212,
        "other_futex_wait_ns": 1950579,
        "other_off_cpu_ns": 3814726512,
        "mutex_wait_per_cpu": 0.00020005624225776511
      },
      "1439232": {
        "comm": "moq-quic-2",
        "switches": 13234,
        "cpu_ns": 1335022680,
        "mutex_wait_ns": 273481,
        "other_futex_wait_ns": 3546554,
        "other_off_cpu_ns": 3689921886,
        "mutex_wait_per_cpu": 0.00020485120147921383
      },
      "1439233": {
        "comm": "moq-quic-3",
        "switches": 14704,
        "cpu_ns": 1156308644,
        "mutex_wait_ns": 587704,
        "other_futex_wait_ns": 1942549,
        "other_off_cpu_ns": 3869576496,
        "mutex_wait_per_cpu": 0.0005082587620957022
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15359.253499222394,
      "recv_fps": 30714.285714285714,
      "send_mbps": 147.44883359253498,
      "recv_mbps": 294.85714285714283,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 0.9869308261947554,
      "cpu_us_per_frame": 32.13263155052692,
      "throughput_window_start_ms": 1793712386969,
      "throughput_window_end_ms": 1793712391470,
      "host_window_start_ms": 1793712386969,
      "host_window_end_ms": 1793712391470,
      "threads": 7,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15357.471494298861,
      "recv_fps": 31481.496299259856,
      "send_mbps": 147.43172634526906,
      "recv_mbps": 302.2223644728946,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 1.0158359017112804,
      "cpu_us_per_frame": 32.2677134547497,
      "throughput_window_start_ms": 1793712397034,
      "throughput_window_end_ms": 1793712402033,
      "host_window_start_ms": 1793712397034,
      "host_window_end_ms": 1793712402033,
      "threads": 7,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 8,
    "group_share_pct": 0.5032291586071371,
    "wait_by_type_us": {
      "TrackState": 1503,
      "other": 16,
      "GroupState": 8
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 199
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 167
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 117
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 106
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;poll_closed;<moq_net::model::track::TrackState>::poll_closed;<kio::weak::ConsumerWeak<moq_net::model::track::TrackState>>::poll_closed;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 103
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 94
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 84
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 74
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 65
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 65
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 58
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Dynamic>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 45
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 38
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 36
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 32
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 27
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Producer>::subscription;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 25
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 24
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 22
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 20
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 16
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 14
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 13
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;poll_moved;<moq_net::model::resume::Consumer>::poll_changed;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#1}::poll_changed::{closure_env#0},;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 5
[unknown];<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Handle>::cache_refresh;read<moq_net::model::group::GroupState>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 5
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_used;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 5
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 4
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 4
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_unused;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_read_frame;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#24}::poll_read_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2

run.AW5tm7, current, 1 workers

{
  "source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5099434551,
    "workers": {
      "1439673": {
        "comm": "moq-quic-0",
        "switches": 8956,
        "cpu_ns": 3214809670,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 1849614646,
        "mutex_wait_per_cpu": 0.0
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15363.6,
      "recv_fps": 31284.4,
      "send_mbps": 147.49056,
      "recv_mbps": 300.33024,
      "loss_pct": 0,
      "latency_p99_ms": 2986,
      "cpu_cores": 0.642874251497006,
      "cpu_us_per_frame": 20.549355317570612,
      "throughput_window_start_ms": 1793712452443,
      "throughput_window_end_ms": 1793712457443,
      "host_window_start_ms": 1793712452443,
      "host_window_end_ms": 1793712457443,
      "threads": 4,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15363.072614522905,
      "recv_fps": 31251.250250050012,
      "send_mbps": 147.4854970994199,
      "recv_mbps": 300.01200240048007,
      "loss_pct": 0,
      "latency_p99_ms": 2986,
      "cpu_cores": 0.6446574105240208,
      "cpu_us_per_frame": 20.628211843236226,
      "throughput_window_start_ms": 1793712463003,
      "throughput_window_end_ms": 1793712468002,
      "host_window_start_ms": 1793712463003,
      "host_window_end_ms": 1793712468002,
      "threads": 4,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 0,
    "group_share_pct": null,
    "wait_by_type_us": {},
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

(empty: no mutex waits recorded)

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 3/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.AW5tm7, current, 16 workers

{
  "source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-10",
      "moq-quic-11",
      "moq-quic-12",
      "moq-quic-13",
      "moq-quic-14",
      "moq-quic-15",
      "moq-quic-2",
      "moq-quic-3",
      "moq-quic-4",
      "moq-quic-5",
      "moq-quic-6",
      "moq-quic-7",
      "moq-quic-8",
      "moq-quic-9"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5098674978,
    "workers": {
      "1439959": {
        "comm": "moq-quic-0",
        "switches": 13057,
        "cpu_ns": 312151867,
        "mutex_wait_ns": 2591738,
        "other_futex_wait_ns": 115667,
        "other_off_cpu_ns": 4695901735,
        "mutex_wait_per_cpu": 0.008302811144166566
      },
      "1439960": {
        "comm": "moq-quic-1",
        "switches": 14140,
        "cpu_ns": 413434648,
        "mutex_wait_ns": 6078633,
        "other_futex_wait_ns": 351407,
        "other_off_cpu_ns": 4591222457,
        "mutex_wait_per_cpu": 0.01470276627613465
      },
      "1439961": {
        "comm": "moq-quic-2",
        "switches": 14209,
        "cpu_ns": 405623508,
        "mutex_wait_ns": 4131563,
        "other_futex_wait_ns": 129125,
        "other_off_cpu_ns": 4602497856,
        "mutex_wait_per_cpu": 0.01018570896043826
      },
      "1439962": {
        "comm": "moq-quic-3",
        "switches": 9788,
        "cpu_ns": 207983834,
        "mutex_wait_ns": 1440708,
        "other_futex_wait_ns": 17625,
        "other_off_cpu_ns": 4799151526,
        "mutex_wait_per_cpu": 0.006927019145151445
      },
      "1439963": {
        "comm": "moq-quic-4",
        "switches": 11282,
        "cpu_ns": 295618025,
        "mutex_wait_ns": 2884921,
        "other_futex_wait_ns": 35937,
        "other_off_cpu_ns": 4710180149,
        "mutex_wait_per_cpu": 0.0097589482238101
      },
      "1439964": {
        "comm": "moq-quic-5",
        "switches": 12837,
        "cpu_ns": 309358982,
        "mutex_wait_ns": 4203357,
        "other_futex_wait_ns": 847283,
        "other_off_cpu_ns": 4696163544,
        "mutex_wait_per_cpu": 0.013587311972729468
      },
      "1439965": {
        "comm": "moq-quic-6",
        "switches": 14486,
        "cpu_ns": 485281227,
        "mutex_wait_ns": 5402492,
        "other_futex_wait_ns": 265905,
        "other_off_cpu_ns": 4522963321,
        "mutex_wait_per_cpu": 0.011132703470517724
      },
      "1439966": {
        "comm": "moq-quic-7",
        "switches": 12504,
        "cpu_ns": 332216066,
        "mutex_wait_ns": 3654192,
        "other_futex_wait_ns": 214889,
        "other_off_cpu_ns": 4674902218,
        "mutex_wait_per_cpu": 0.010999443958258177
      },
      "1439967": {
        "comm": "moq-quic-8",
        "switches": 11867,
        "cpu_ns": 314118378,
        "mutex_wait_ns": 4006445,
        "other_futex_wait_ns": 577540,
        "other_off_cpu_ns": 4691839913,
        "mutex_wait_per_cpu": 0.012754570507810275
      },
      "1439968": {
        "comm": "moq-quic-9",
        "switches": 12897,
        "cpu_ns": 352184693,
        "mutex_wait_ns": 2617438,
        "other_futex_wait_ns": 18055,
        "other_off_cpu_ns": 4656279998,
        "mutex_wait_per_cpu": 0.007432003866221409
      },
      "1439969": {
        "comm": "moq-quic-10",
        "switches": 12533,
        "cpu_ns": 293130016,
        "mutex_wait_ns": 2405221,
        "other_futex_wait_ns": 168912,
        "other_off_cpu_ns": 4714726707,
        "mutex_wait_per_cpu": 0.008205304365691434
      },
      "1439970": {
        "comm": "moq-quic-11",
        "switches": 13838,
        "cpu_ns": 394295249,
        "mutex_wait_ns": 4767638,
        "other_futex_wait_ns": 41736,
        "other_off_cpu_ns": 4612918285,
        "mutex_wait_per_cpu": 0.012091543106571899
      },
      "1439971": {
        "comm": "moq-quic-12",
        "switches": 14274,
        "cpu_ns": 429488658,
        "mutex_wait_ns": 4846517,
        "other_futex_wait_ns": 76804,
        "other_off_cpu_ns": 4578043504,
        "mutex_wait_per_cpu": 0.011284388795198405
      },
      "1439972": {
        "comm": "moq-quic-13",
        "switches": 10866,
        "cpu_ns": 270156893,
        "mutex_wait_ns": 2894176,
        "other_futex_wait_ns": 49352,
        "other_off_cpu_ns": 4736654137,
        "mutex_wait_per_cpu": 0.010712945236603679
      },
      "1439973": {
        "comm": "moq-quic-14",
        "switches": 14395,
        "cpu_ns": 456332944,
        "mutex_wait_ns": 7024446,
        "other_futex_wait_ns": 109072,
        "other_off_cpu_ns": 4549574201,
        "mutex_wait_per_cpu": 0.015393247610893506
      },
      "1439974": {
        "comm": "moq-quic-15",
        "switches": 9847,
        "cpu_ns": 217682243,
        "mutex_wait_ns": 2276747,
        "other_futex_wait_ns": 3698,
        "other_off_cpu_ns": 4789566817,
        "mutex_wait_per_cpu": 0.010459038682360509
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15360,
      "recv_fps": 31149.2,
      "send_mbps": 147.456,
      "recv_mbps": 299.03232,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 1.080877511908454,
      "cpu_us_per_frame": 34.70000872922752,
      "throughput_window_start_ms": 1793712508934,
      "throughput_window_end_ms": 1793712513934,
      "host_window_start_ms": 1793712508934,
      "host_window_end_ms": 1793712513934,
      "threads": 19,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15360,
      "recv_fps": 30723.8,
      "send_mbps": 147.456,
      "recv_mbps": 294.94848,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 1.1929561752988045,
      "cpu_us_per_frame": 38.828405838431586,
      "throughput_window_start_ms": 1793712519998,
      "throughput_window_end_ms": 1793712524998,
      "host_window_start_ms": 1793712519998,
      "host_window_end_ms": 1793712524998,
      "threads": 19,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 937,
    "group_share_pct": 1.530389784561624,
    "wait_by_type_us": {
      "TrackState": 57019,
      "Waiters": 2371,
      "GroupState": 937,
      "other": 236
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Requests>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 12141
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#29}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 11757
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8163
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3208
<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::lite::subscriber::TrackEntry>::open;is_live;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3179
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2845
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2827
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2639
<moq_net::lite::subscriber::UniAccept>::poll;poll<moq_net::lite::subscriber::UniServe>;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Request>::subscription;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2476
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2127
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_read_frame<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;park_value<moq_net::model::group::GroupState>;waiters<moq_net::model::group::GroupState>;lock<kio::channel::Waiters>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1636
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1509
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 867
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll_finished;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 849
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 786
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;take;take;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Cursor>::prune_skipped;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 635
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_next_frame;poll<moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::option::Option<moq_net::model::group::Source>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;park_value<moq_net::model::group::GroupState>;waiters<moq_net::model::group::GroupState>;lock<kio::channel::Waiters>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 625
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_read_frame<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 561
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Producer>::poll_subscription_changed;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 512
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 244
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Handle>::cache_refresh;read<moq_net::model::group::GroupState>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 214
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;take;take;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 163
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;<moq_net::model::resume::Recover>::wants;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#15}::wants::{closure_env#1},;repoll<moq_net::model::resume::Route>;<kio::channel::Channel<moq_net::model::resume::Route>>::park_value_cold;park_value<moq_net::model::resume::Route>;waiters<moq_net::model::resume::Route>;lock<kio::channel::Waiters>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 110
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_next_frame;poll<moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::option::Option<moq_net::model::group::Source>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 109
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 108
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 92
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#29}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#29}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 89
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_expiry;expiry_pending;expiry_pending;read<moq_net::model::group::GroupState>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 53
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 19
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#29}::poll_finished::{closure_env#0}>;<moq_net::model::resume::Recover>::wants;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#15}::wants::{closure_env#1},;poll_judged<moq_net::model::resume::Route, moq_net::model::resume::{impl#15}::wants::{closure_env#1},;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 4
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_end<moq_net::model::datagram::Stored, moq_net::model::resume::{impl#13}::poll_datagram_once::{closure_env#1}>;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#29}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#29}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#29}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 4/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.AW5tm7, current, 4 workers

{
  "source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-2",
      "moq-quic-3"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1~1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5096001824,
    "workers": {
      "1439789": {
        "comm": "moq-quic-0",
        "switches": 14585,
        "cpu_ns": 1125106802,
        "mutex_wait_ns": 448072,
        "other_futex_wait_ns": 368531,
        "other_off_cpu_ns": 3898504227,
        "mutex_wait_per_cpu": 0.0003982484144647452
      },
      "1439790": {
        "comm": "moq-quic-1",
        "switches": 15488,
        "cpu_ns": 1056182184,
        "mutex_wait_ns": 480357,
        "other_futex_wait_ns": 820992,
        "other_off_cpu_ns": 3966620563,
        "mutex_wait_per_cpu": 0.00045480505851819975
      },
      "1439791": {
        "comm": "moq-quic-2",
        "switches": 15449,
        "cpu_ns": 1041876103,
        "mutex_wait_ns": 857504,
        "other_futex_wait_ns": 772572,
        "other_off_cpu_ns": 3978996898,
        "mutex_wait_per_cpu": 0.0008230383608289747
      },
      "1439792": {
        "comm": "moq-quic-3",
        "switches": 15126,
        "cpu_ns": 1050983243,
        "mutex_wait_ns": 507236,
        "other_futex_wait_ns": 834966,
        "other_off_cpu_ns": 3971085378,
        "mutex_wait_per_cpu": 0.0004826299594959384
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15360.472094418885,
      "recv_fps": 31135.027005401083,
      "send_mbps": 147.46053210642128,
      "recv_mbps": 298.8962592518504,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 0.8543265539335413,
      "cpu_us_per_frame": 27.43940301658768,
      "throughput_window_start_ms": 1793712477461,
      "throughput_window_end_ms": 1793712482460,
      "host_window_start_ms": 1793712477461,
      "host_window_end_ms": 1793712482460,
      "threads": 7,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15356.587425016662,
      "recv_fps": 30710.064430126637,
      "send_mbps": 147.42323928015995,
      "recv_mbps": 294.8166185292157,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 0.8780154762203788,
      "cpu_us_per_frame": 28.590479782876773,
      "throughput_window_start_ms": 1793712488520,
      "throughput_window_end_ms": 1793712493021,
      "host_window_start_ms": 1793712488520,
      "host_window_end_ms": 1793712493021,
      "threads": 7,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 26,
    "group_share_pct": 1.1338021750686496,
    "wait_by_type_us": {
      "TrackState": 2041,
      "other": 146,
      "GroupState": 26,
      "Waiters": 9
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Requests>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 499
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#29}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 331
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 234
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 223
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 222
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 167
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;poll_moved;<moq_net::model::resume::Consumer>::poll_changed;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#5}::poll_changed::{closure_env#0},;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#5}::poll_changed::{closure_env#0},;poll_judged<moq_net::model::resume::Route, moq_net::model::resume::{impl#5}::poll_changed::{closure_env#0},;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 143
<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::lite::subscriber::TrackEntry>::open;is_live;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 85
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 82
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 64
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 51
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll_finished;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 31
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_read_frame<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 26
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;take;take;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Cursor>::prune_skipped;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 14
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 12
<moq_net::lite::subscriber::UniAccept>::poll;poll<moq_net::lite::subscriber::UniServe>;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Request>::subscription;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_next_frame;poll<moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::option::Option<moq_net::model::group::Source>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_next_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Source>, moq_net::error::Error>>;park_value<moq_net::model::group::GroupState>;waiters<moq_net::model::group::GroupState>;lock<kio::channel::Waiters>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 5
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_read_frame<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;park_value<moq_net::model::group::GroupState>;waiters<moq_net::model::group::GroupState>;lock<kio::channel::Waiters>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 4
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Producer>::poll_subscription_changed;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;read<moq_net::model::resume::Route>;lock<moq_net::model::resume::Route>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3

run.nLyO6T, base, 1 workers

{
  "source_revision": "d854bf98b1a9daee5c65a841da2d655ed6613c6d",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5097329362,
    "workers": {
      "1447366": {
        "comm": "moq-quic-0",
        "switches": 5949,
        "cpu_ns": 3830091889,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 1243132024,
        "mutex_wait_per_cpu": 0.0
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15363.072614522905,
      "recv_fps": 31489.89797959592,
      "send_mbps": 147.4854970994199,
      "recv_mbps": 302.30302060412083,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 0.7644562844704671,
      "cpu_us_per_frame": 24.27623884224082,
      "throughput_window_start_ms": 1793712881161,
      "throughput_window_end_ms": 1793712886160,
      "host_window_start_ms": 1793712881161,
      "host_window_end_ms": 1793712886160,
      "threads": 4,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15360,
      "recv_fps": 30719.8,
      "send_mbps": 147.456,
      "recv_mbps": 294.91008,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 0.7613293413173653,
      "cpu_us_per_frame": 24.783017510444903,
      "throughput_window_start_ms": 1793712892235,
      "throughput_window_end_ms": 1793712897235,
      "host_window_start_ms": 1793712892235,
      "host_window_end_ms": 1793712897235,
      "threads": 4,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 0,
    "group_share_pct": null,
    "wait_by_type_us": {},
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

(empty: no mutex waits recorded)

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 5/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.nLyO6T, base, 16 workers

{
  "source_revision": "d854bf98b1a9daee5c65a841da2d655ed6613c6d",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-10",
      "moq-quic-11",
      "moq-quic-12",
      "moq-quic-13",
      "moq-quic-14",
      "moq-quic-15",
      "moq-quic-2",
      "moq-quic-3",
      "moq-quic-4",
      "moq-quic-5",
      "moq-quic-6",
      "moq-quic-7",
      "moq-quic-8",
      "moq-quic-9"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5100214009,
    "workers": {
      "1447609": {
        "comm": "moq-quic-0",
        "switches": 12582,
        "cpu_ns": 367909430,
        "mutex_wait_ns": 2030827,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4643126390,
        "mutex_wait_per_cpu": 0.005519910158323476
      },
      "1447610": {
        "comm": "moq-quic-1",
        "switches": 13029,
        "cpu_ns": 442406806,
        "mutex_wait_ns": 2733298,
        "other_futex_wait_ns": 426982,
        "other_off_cpu_ns": 4569810799,
        "mutex_wait_per_cpu": 0.006178245820205578
      },
      "1447611": {
        "comm": "moq-quic-2",
        "switches": 10641,
        "cpu_ns": 276875531,
        "mutex_wait_ns": 1397843,
        "other_futex_wait_ns": 29331,
        "other_off_cpu_ns": 4733831864,
        "mutex_wait_per_cpu": 0.005048633206955366
      },
      "1447612": {
        "comm": "moq-quic-3",
        "switches": 13013,
        "cpu_ns": 477870944,
        "mutex_wait_ns": 3009241,
        "other_futex_wait_ns": 37945,
        "other_off_cpu_ns": 4534751106,
        "mutex_wait_per_cpu": 0.006297183450433848
      },
      "1447613": {
        "comm": "moq-quic-4",
        "switches": 14033,
        "cpu_ns": 466309275,
        "mutex_wait_ns": 2841866,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4546718047,
        "mutex_wait_per_cpu": 0.006094380172901343
      },
      "1447614": {
        "comm": "moq-quic-5",
        "switches": 13488,
        "cpu_ns": 415669459,
        "mutex_wait_ns": 2428825,
        "other_futex_wait_ns": 203020,
        "other_off_cpu_ns": 4596220338,
        "mutex_wait_per_cpu": 0.005843164436095845
      },
      "1447615": {
        "comm": "moq-quic-6",
        "switches": 13360,
        "cpu_ns": 411080022,
        "mutex_wait_ns": 2107329,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4601282432,
        "mutex_wait_per_cpu": 0.005126323069039828
      },
      "1447616": {
        "comm": "moq-quic-7",
        "switches": 14279,
        "cpu_ns": 551859031,
        "mutex_wait_ns": 2998012,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4462186956,
        "mutex_wait_per_cpu": 0.0054325685212896335
      },
      "1447617": {
        "comm": "moq-quic-8",
        "switches": 13042,
        "cpu_ns": 449841059,
        "mutex_wait_ns": 1417434,
        "other_futex_wait_ns": 234169,
        "other_off_cpu_ns": 4563985174,
        "mutex_wait_per_cpu": 0.0031509662616190843
      },
      "1447618": {
        "comm": "moq-quic-9",
        "switches": 11172,
        "cpu_ns": 344089129,
        "mutex_wait_ns": 2866963,
        "other_futex_wait_ns": 176619,
        "other_off_cpu_ns": 4666245849,
        "mutex_wait_per_cpu": 0.008332035970831268
      },
      "1447619": {
        "comm": "moq-quic-10",
        "switches": 11885,
        "cpu_ns": 348646788,
        "mutex_wait_ns": 1294563,
        "other_futex_wait_ns": 34302,
        "other_off_cpu_ns": 4663559052,
        "mutex_wait_per_cpu": 0.0037131074903234158
      },
      "1447620": {
        "comm": "moq-quic-11",
        "switches": 12264,
        "cpu_ns": 417542766,
        "mutex_wait_ns": 2436230,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4595220330,
        "mutex_wait_per_cpu": 0.005834683769853649
      },
      "1447621": {
        "comm": "moq-quic-12",
        "switches": 11114,
        "cpu_ns": 327880851,
        "mutex_wait_ns": 513830,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 4684851268,
        "mutex_wait_per_cpu": 0.0015671241502298041
      },
      "1447622": {
        "comm": "moq-quic-13",
        "switches": 8434,
        "cpu_ns": 218222198,
        "mutex_wait_ns": 809559,
        "other_futex_wait_ns": 22278,
        "other_off_cpu_ns": 4790937571,
        "mutex_wait_per_cpu": 0.0037097921633068693
      },
      "1447623": {
        "comm": "moq-quic-14",
        "switches": 12514,
        "cpu_ns": 417776679,
        "mutex_wait_ns": 2294864,
        "other_futex_wait_ns": 74032,
        "other_off_cpu_ns": 4594525481,
        "mutex_wait_per_cpu": 0.005493039978902221
      },
      "1447624": {
        "comm": "moq-quic-15",
        "switches": 14286,
        "cpu_ns": 531607146,
        "mutex_wait_ns": 2541566,
        "other_futex_wait_ns": 724767,
        "other_off_cpu_ns": 4481733595,
        "mutex_wait_per_cpu": 0.004780910149766874
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15360.672134426886,
      "recv_fps": 30725.745149029808,
      "send_mbps": 147.4624524904981,
      "recv_mbps": 294.96715343068615,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 1.2494674583619318,
      "cpu_us_per_frame": 40.66516376744031,
      "throughput_window_start_ms": 1793712940414,
      "throughput_window_end_ms": 1793712945413,
      "host_window_start_ms": 1793712940414,
      "host_window_end_ms": 1793712945413,
      "threads": 19,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15356.4,
      "recv_fps": 31440.8,
      "send_mbps": 147.42144,
      "recv_mbps": 301.83168,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 1.3718747286303885,
      "cpu_us_per_frame": 43.633582117197676,
      "throughput_window_start_ms": 1793712950472,
      "throughput_window_end_ms": 1793712955472,
      "host_window_start_ms": 1793712950472,
      "host_window_end_ms": 1793712955472,
      "threads": 19,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 599,
    "group_share_pct": 1.77627530784571,
    "wait_by_type_us": {
      "TrackState": 32128,
      "GroupState": 599,
      "other": 514
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2079
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1825
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1818
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1563
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1414
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1411
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1171
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1112
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1112
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1105
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 977
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 944
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;poll_closed;<moq_net::model::track::TrackState>::poll_closed;<kio::weak::ConsumerWeak<moq_net::model::track::TrackState>>::poll_closed;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 939
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 938
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 937
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 932
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 901
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 855
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 736
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 621
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 605
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_read_frame;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#25}::poll_read_frame::{closure_env#2}, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 599
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 597
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 587
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 565
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_used;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 544
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 508
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 472
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 460
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;next_filled;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 430
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 427
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 401
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Dynamic>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 362
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 351
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 319
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 276
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 266
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 238
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 224
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 215
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 194
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::frame::Consumer>::poll_read_chunk;poll_resumed<core::option::Option<bytes::bytes::Bytes>, fn(&mut moq_net::model::frame::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<bytes::bytes::Bytes>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::frame::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<bytes::bytes::Bytes>, moq_net::error::Error>>,;<moq_net::model::frame::Consumer>::poll_read_chunk_once;<moq_net::model::frame::Consumer>::poll_expired;and_then<&alloc::boxed::Box<(moq_net::model::resume::Recover, u64), alloc::alloc::Global>, core::time::Duration, moq_net::model::frame::{impl#19}::poll_expired::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 183
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#24}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 169
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Producer>::poll_subscription_changed;<kio::weak::ConsumerWeak<moq_net::model::track::TrackState>>::poll_closed;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 158
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 147
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;subscriptions;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 121
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Producer>::subscription;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 110
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 101
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;poll<moq_net::model::track::{impl#35}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 71
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#24}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 34
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;is_live;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 31
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 29
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::track::Subscriber>::poll_group;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 24
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_end<moq_net::model::resume::Candidate, moq_net::model::resume::{impl#5}::poll_lowest::{closure_env#2}>;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_unused;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;poll_moved;<moq_net::model::resume::Consumer>::poll_changed;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#1}::poll_changed::{closure_env#0},;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;<moq_net::model::resume::Recover>::wants;<moq_net::model::resume::Consumer>::poll_changed;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#1}::poll_changed::{closure_env#0},;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 6/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.nLyO6T, base, 4 workers

{
  "source_revision": "d854bf98b1a9daee5c65a841da2d655ed6613c6d",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-2",
      "moq-quic-3"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5099898846,
    "workers": {
      "1447454": {
        "comm": "moq-quic-0",
        "switches": 13311,
        "cpu_ns": 1352278449,
        "mutex_wait_ns": 2429281,
        "other_futex_wait_ns": 722260,
        "other_off_cpu_ns": 3674915004,
        "mutex_wait_per_cpu": 0.0017964354913712005
      },
      "1447455": {
        "comm": "moq-quic-1",
        "switches": 14540,
        "cpu_ns": 1232099355,
        "mutex_wait_ns": 2292139,
        "other_futex_wait_ns": 183406,
        "other_off_cpu_ns": 3792063839,
        "mutex_wait_per_cpu": 0.0018603524064015114
      },
      "1447456": {
        "comm": "moq-quic-2",
        "switches": 13904,
        "cpu_ns": 1121990828,
        "mutex_wait_ns": 1652083,
        "other_futex_wait_ns": 231686,
        "other_off_cpu_ns": 3902942878,
        "mutex_wait_per_cpu": 0.0014724567784078178
      },
      "1447457": {
        "comm": "moq-quic-3",
        "switches": 13076,
        "cpu_ns": 1360513206,
        "mutex_wait_ns": 1238948,
        "other_futex_wait_ns": 902358,
        "other_off_cpu_ns": 3666979118,
        "mutex_wait_per_cpu": 0.0009106475369265912
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15362.4,
      "recv_fps": 31682.6,
      "send_mbps": 147.47904,
      "recv_mbps": 304.15296,
      "loss_pct": 0,
      "latency_p99_ms": 3985,
      "cpu_cores": 0.9860039920159681,
      "cpu_us_per_frame": 31.121309236488422,
      "throughput_window_start_ms": 1793712906090,
      "throughput_window_end_ms": 1793712911090,
      "host_window_start_ms": 1793712906090,
      "host_window_end_ms": 1793712911090,
      "threads": 7,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15362.872574514904,
      "recv_fps": 30726.14522904581,
      "send_mbps": 147.48357671534308,
      "recv_mbps": 294.9709941988398,
      "loss_pct": 0,
      "latency_p99_ms": 3984,
      "cpu_cores": 1.028002007587146,
      "cpu_us_per_frame": 33.456914296407184,
      "throughput_window_start_ms": 1793712917173,
      "throughput_window_end_ms": 1793712922172,
      "host_window_start_ms": 1793712917173,
      "host_window_end_ms": 1793712922172,
      "threads": 7,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 0,
    "group_share_pct": 0.0,
    "wait_by_type_us": {
      "TrackState": 7368,
      "other": 44
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1445
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 673
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Dynamic>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 635
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 441
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 412
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_used;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 367
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 326
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 301
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 259
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 232
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 220
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 211
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 209
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 189
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 152
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 137
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 125
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 125
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 121
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 114
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 99
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 75
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 73
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 69
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 54
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;poll_closed;<moq_net::model::track::TrackState>::poll_closed;<kio::weak::ConsumerWeak<moq_net::model::track::TrackState>>::poll_closed;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 50
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 44
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 32
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 28
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;next_filled;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 21
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 21
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 19
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 15
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 14
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Producer>::subscription;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 14
<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 13
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 12
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;<moq_net::model::resume::Recover>::wants;<moq_net::model::resume::Consumer>::poll_changed;poll<moq_net::model::resume::Route, moq_net::model::resume::{impl#1}::poll_changed::{closure_env#0},;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 7
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#24}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#24}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;is_used;is_closed<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 5
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#24}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_end<moq_net::model::resume::Candidate, moq_net::model::resume::{impl#5}::poll_lowest::{closure_env#2}>;<moq_net::model::resume::Consumer>::end;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 2

run.nLyO6T, current, 1 workers

{
  "source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5104341396,
    "workers": {
      "1447751": {
        "comm": "moq-quic-0",
        "switches": 7756,
        "cpu_ns": 3288776605,
        "mutex_wait_ns": 0,
        "other_futex_wait_ns": 0,
        "other_off_cpu_ns": 1780610410,
        "mutex_wait_per_cpu": 0.0
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15354.143523661407,
      "recv_fps": 30717.840479893355,
      "send_mbps": 147.3997778271495,
      "recv_mbps": 294.8912686069762,
      "loss_pct": 0,
      "latency_p99_ms": 2986,
      "cpu_cores": 0.658682634730539,
      "cpu_us_per_frame": 21.442999391890385,
      "throughput_window_start_ms": 1793712983964,
      "throughput_window_end_ms": 1793712988465,
      "host_window_start_ms": 1793712983964,
      "host_window_end_ms": 1793712988465,
      "threads": 4,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15363,
      "recv_fps": 31128,
      "send_mbps": 147.4848,
      "recv_mbps": 298.8288,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 0.6548143712574848,
      "cpu_us_per_frame": 21.03618514705361,
      "throughput_window_start_ms": 1793712994043,
      "throughput_window_end_ms": 1793712999043,
      "host_window_start_ms": 1793712994043,
      "host_window_end_ms": 1793712999043,
      "threads": 4,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 0,
    "group_share_pct": null,
    "wait_by_type_us": {},
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

(empty: no mutex waits recorded)

(Written by GPT-6.1 Sol)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, part 7/7. Full acquisition stacks and validated lock/runtime summaries from the unchanged profiling recipe. run.AW5tm7 is the pre-epoch comparison; run.nLyO6T is the epoch comparison. A current 16-worker capture in the second run failed and is excluded.

run.nLyO6T, current, 4 workers

{
  "source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "current_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "harness_source_revision": "b5a2e8fb7716144abd756ae81cd9a780b200be0b",
  "runtime": {
    "runtime": "tokio-workers",
    "workers": [
      "moq-quic-0",
      "moq-quic-1",
      "moq-quic-2",
      "moq-quic-3"
    ]
  },
  "rustc": "rustc 1.98.1 (48a229cea 2026-09-01)\nbinary: rustc\ncommit-hash: 48a229ceaefd4985c50990b14116b6d856af0985\ncommit-date: 2026-09-01\nhost: x86_64-unknown-linux-gnu\nrelease: 1.98.1\nLLVM version: 22.1.8",
  "cargo": "cargo 1.98.1 (797e8a9bc 2026-08-05)",
  "affinity": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7
  ],
  "profile": "profiling",
  "features": "default + moq-relay/io-uring,moq-bench/uring; heap adds moq-relay/jemalloc",
  "environment": {
    "BENCH_PROFILE_TRANSPORT": "quic",
    "BENCH_PROFILE_MODES": "lock",
    "RUSTFLAGS": "-C force-frame-pointers=yes",
    "BENCH_PROFILE_WORKERS": "1 4 16",
    "BENCH_PROFILE_WORKLOAD": "video-heavy",
    "BENCH_BASE": "d854bf98b1",
    "BENCH_PROFILE_RUNTIME": "tokio-workers"
  },
  "source_dirty_patch_bytes": 0,
  "untracked_sources": [],
  "profiler": {
    "perf": "perf version 7.3.0",
    "jeprof": "jeprof (part of jemalloc 5.4.0-0-g0000000000000000000000000000000000000000)"
  },
  "lock": {
    "version": 3,
    "window_ns": 5098959050,
    "workers": {
      "1447910": {
        "comm": "moq-quic-0",
        "switches": 15357,
        "cpu_ns": 1080233290,
        "mutex_wait_ns": 1094352,
        "other_futex_wait_ns": 455116,
        "other_off_cpu_ns": 3943538391,
        "mutex_wait_per_cpu": 0.0010130700563764333
      },
      "1447911": {
        "comm": "moq-quic-1",
        "switches": 14223,
        "cpu_ns": 1149499578,
        "mutex_wait_ns": 772379,
        "other_futex_wait_ns": 717408,
        "other_off_cpu_ns": 3874390646,
        "mutex_wait_per_cpu": 0.0006719263014814261
      },
      "1447912": {
        "comm": "moq-quic-2",
        "switches": 14539,
        "cpu_ns": 1071131502,
        "mutex_wait_ns": 592793,
        "other_futex_wait_ns": 58850,
        "other_off_cpu_ns": 3952046518,
        "mutex_wait_per_cpu": 0.0005534269124688669
      },
      "1447913": {
        "comm": "moq-quic-3",
        "switches": 16010,
        "cpu_ns": 960720128,
        "mutex_wait_ns": 800453,
        "other_futex_wait_ns": 71264,
        "other_off_cpu_ns": 4060059615,
        "mutex_wait_per_cpu": 0.0008331802120835757
      }
    },
    "idle": [],
    "scope": "exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded"
  },
  "summaries": {
    "control": {
      "send_fps": 15361.030882026214,
      "recv_fps": 30709.397911575204,
      "send_mbps": 147.46589646745167,
      "recv_mbps": 294.81021995112195,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 0.8582834331337323,
      "cpu_us_per_frame": 27.948559447667385,
      "throughput_window_start_ms": 1793713007742,
      "throughput_window_end_ms": 1793713012243,
      "host_window_start_ms": 1793713007742,
      "host_window_end_ms": 1793713012243,
      "threads": 7,
      "udp_drops": 0
    },
    "profiled": {
      "send_fps": 15361.253054876694,
      "recv_fps": 30720.506554099087,
      "send_mbps": 147.46802932681626,
      "recv_mbps": 294.9168629193512,
      "loss_pct": 0,
      "latency_p99_ms": 2985,
      "cpu_cores": 0.8759818731787702,
      "cpu_us_per_frame": 28.514564746390437,
      "throughput_window_start_ms": 1793713018303,
      "throughput_window_end_ms": 1793713022804,
      "host_window_start_ms": 1793713018303,
      "host_window_end_ms": 1793713022804,
      "threads": 7,
      "udp_drops": 0
    }
  },
  "attribution": {
    "method": "Innermost lock<T> frame identifies the mutex. Waiters locks are separate even when outer frames mention GroupState.",
    "group_wait_us": 18,
    "group_share_pct": 0.5521511348086198,
    "wait_by_type_us": {
      "TrackState": 3148,
      "GroupState": 18
    },
    "rounding": "Folded wait weights truncate each interval to microseconds; lock.json totals retain nanoseconds."
  }
}

Full mutex.folded (weights in microseconds):

<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Requests>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 879
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 478
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 263
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 256
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 254
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 227
<moq_net::lite::subscriber::UniAccept>::poll;poll<moq_net::lite::subscriber::UniServe>;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Request>::subscription;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 204
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#29}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 186
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Producer>::poll_subscription_changed;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 119
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;poll_recv_with;poll<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 74
<moq_net::model::track::Subscriber>::poll_peek_group;poll_peek_group;poll_lowest;<moq_net::model::track::Subscriber>::poll_peek_group;<moq_net::model::track::Cursor>::poll_peek;poll_peek_recv;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 46
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 38
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll::{closure_env#0}<moq_net::model::track::{impl#46}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 32
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;subscriptions;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 26
<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll::{closure#0}>;passes<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;pass<moq_net::lite::subscriber::UniServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::subscriber::UniServe>>;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::lite::subscriber::TrackEntry>::open;is_live;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 23
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_drop::{closure_env#0},;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 20
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll<moq_net::lite::publisher::GroupServe>;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll_each::<<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll::{closure#0}>;passes<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;pass<moq_net::lite::publisher::GroupServe, kio::task::{impl#6}::poll::{closure_env#0}<moq_net::lite::publisher::GroupServe>>;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;<moq_net::model::group::Consumer>::poll_read_object;poll_read_cached;poll_read_as<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_through::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_resumed<moq_net::model::frame::Cached, moq_net::model::group::{impl#29}::poll_read_frame_through::{closure_env#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>>;{closure#0}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;<moq_net::model::group::Consumer>::poll_read_frame_once::<moq_net::model::frame::Cached, <moq_net::model::group::Prefetch>::pop_cached>;poll_read_frame<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;poll_judged<moq_net::model::group::GroupState, moq_net::model::group::{impl#30}::poll_read_frame::{closure_env#2}<moq_net::model::frame::Cached, fn(&mut moq_net::model::group::Prefetch) -> core::option::Option<moq_net::model::frame::Cached>>, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;read<moq_net::model::group::GroupState>;lock<moq_net::model::group::GroupState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 18
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;take;take;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Subscriber>::forget_past_budget;forget_past_budget;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 9
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Subscriber>::take;take;take;<moq_net::model::track::Subscriber>::take;hand_out;<moq_net::model::track::Cursor>::prune_skipped;read<moq_net::model::track::TrackState>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_track;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll_finished;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;poll_judged<moq_net::model::track::TrackState, moq_net::model::track::{impl#46}::poll_finished::{closure_env#1}, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;read<moq_net::model::track::TrackState>;lock<moq_net::model::track::TrackState>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 6

Attempt preflight and limitations

{
  "utc": "2026-11-03T13:16:33Z",
  "hostname": "desktop",
  "cpus": 24,
  "load": [
    0.212890625,
    1.0322265625,
    2.3623046875
  ],
  "disk": {
    "total": 1998293323776,
    "used": 1986059730944,
    "free": 12233592832
  },
  "perf_event_paranoid": "-1",
  "performance_cpus": "0-7",
  "nix_shell": "impure",
  "t3_running_threads": "Only this implementation child, verified via t3_thread_list allProjects=true includeSubagents=true."
}

The preflight establishes the starting conditions only. No continuous independent host-idle trace was collected. The historical observations are candidate evidence, not proof that the host stayed quiet throughout the captures. Two all-project T3 checks showed only this implementation child running; other agent systems were not independently audited throughout.

Refused second-current 16-worker capture

Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.
profile: worker moq-quic-0 (1448077) has nonalternating switch records

The recipe exited 1. No accepted profile or lock summary is reported for this capture.

Build-tool automatic GC incident

The first recipe invoked the installed Cargo shim; it automatically collected managed build output and shared cache data before the workload. This violated the no-shared-GC constraint despite no explicit GC command. The second recipe and local checks bypassed the shim using direct pinned Nix Cargo/rustc and BASH_ENV=/dev/null. The GC log lines are retained here to disclose the side effect:

mbx[gc]: 11.2 GiB free on the disk holding /home/kixelated/.cache/mbx, under the 50.0 GiB minimum; collecting private state and managed targets past their budgets, then shared cache objects if the cache disk is short
mbx[gc]: removed 7 target directories (7.4 GiB logical, 6 abandoned and 1 live); 8.3 GiB logical remain
mbx[gc]: evicted 5284 objects and 110 action results (5.4 GiB logical); 90.0 GiB logical remain
mbx[gc]: removed 0 learned incremental directories (0 B logical); 9.7 GiB logical remain
mbx[gc]: kept 4 learned incremental directories with unreadable checkout records
mbx[gc]: removed 3 generated source trees (321 B logical); 13.8 MiB logical remain
mbx[gc]: evicted 34170 shared cache objects and 6751 action results below gc.max_size because the disk was under gc.min_free_size (28.3 GiB logical freed)
mbx[gc]: evicted 756 shared cache objects and 0 action results below gc.max_size because the disk was under gc.min_free_size (957.0 MiB logical freed)
mbx[gc]: evicted 4 shared cache objects and 0 action results below gc.max_size because the disk was under gc.min_free_size (50.0 MiB logical freed)
mbx[gc]: evicted 4 shared cache objects and 0 action results below gc.max_size because the disk was under gc.min_free_size (113.9 MiB logical freed)
mbx[gc]: 49.1 GiB free on the disk holding /home/kixelated/.cache/mbx, under the 50.0 GiB minimum; collecting private state and managed targets past their budgets, then shared cache objects if the cache disk is short
mbx[gc]: evicted 0 objects and 0 action results (3.7 KiB logical); 62.2 GiB logical remain
mbx[gc]: removed 0 learned incremental directories (0 B logical); 9.7 GiB logical remain
mbx[gc]: kept 4 learned incremental directories with unreadable checkout records
mbx[gc]: evicted 1192 shared cache objects and 0 action results below gc.max_size because the disk was under gc.min_free_size (1.2 GiB logical freed)

(Written by GPT-6.1 Sol)

@kixelated kixelated changed the title Log lock profile docs(quest): record mixed log lock profile evidence Nov 3, 2026
@kixelated kixelated mentioned this pull request Nov 3, 2026
Merged
@kixelated @claude
docs(quest): record the nixos2 log lock repeat
1b9f499 
The repeat found no quiet window. Its one completed pre-epoch run shows the
16-worker GroupState wait on the same stack as the epoch capture, so that
wait predates #6782; no valid epoch capture exists on that host yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, nixos2 repeat, part 1/2. Run r1A (run.5e4MLK): unchanged just bench-profile lock video-heavy "1 4 16" tokio-workers d854bf98b1~1, base 7237beb17a, current and harness b5a2e8fb77, on nixos2 (i7-8700, 12 threads, affinity 0-11, perf_event_paranoid=-1, Nix rustc/cargo 1.98.1, perf 7.3.0). Each row is reduced from summary.json, lock.json and mutex.folded; the host fields come from a 2 s monitor over the window from the control's first file to the profiled run's last file. foreign_cores is CPU used by processes outside the run's process tree; agent_tools lists other agents' running tool commands. The current 16-worker capture was refused, so the recipe exited 1 after it.

{"run": "r1A", "label": "base", "workers": 1, "window": [1793732480.0, 1793732508.3], "samples": 16, "load1_max": 3.67, "load1_at_start": 2.4, "busy_cores_max": 3.595, "foreign_cores_mean": 0.07, "foreign_cores_max": 0.138, "kernel_thread_cores_max": 0.108, "agent_processes_max": 4, "agent_tools": [".gh-wrapped", "bash", "codex", "codex-code-mode", "git", "git-remote-http", "quest"], "agent_cores_max": 0.044, "foreign_builds": [], "top_foreign": [".claude-wrapped", "MainThread", "codex", "nsncd"], "control_fps": 22069.8, "control_loss_pct": 37.13474025974026, "profiled_fps": 23220.644128825767, "profiled_loss_pct": 37.2093023255814, "mutex_ms": 0.0, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": null, "by_lock_us": {}, "groupstate_stacks": {}}
{"run": "r1A", "label": "base", "workers": 4, "window": [1793732508.4, 1793732553.7], "samples": 25, "load1_max": 5.21, "load1_at_start": 2.85, "busy_cores_max": 6.549, "foreign_cores_mean": 0.061, "foreign_cores_max": 0.162, "kernel_thread_cores_max": 0.074, "agent_processes_max": 4, "agent_tools": ["bash", "codex", "codex-code-mode", "quest"], "agent_cores_max": 0.088, "foreign_builds": [], "top_foreign": [".claude-wrapped", ".kgx-wrapped", "MainThread", "codex", "nsncd"], "control_fps": 31483.296659331867, "control_loss_pct": 0, "profiled_fps": 30681.18195956454, "profiled_loss_pct": 0, "mutex_ms": 0.822, "idle_workers": 0, "groupstate_us": 9, "groupstate_share_pct": 1.1, "by_lock_us": {"lock<kio::State<moq_net::model::track::TrackState>>": 786, "lock<kio::State<moq_net::model::group::GroupState>>": 9}, "groupstate_stacks": {"<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;expiry_pending;expiry_pending;read<moq_net::model::group::GroupState>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall": 9}}
{"run": "r1A", "label": "base", "workers": 16, "window": [1793732553.7, 1793732638.6], "samples": 43, "load1_max": 12.77, "load1_at_start": 4.88, "busy_cores_max": 8.817, "foreign_cores_mean": 0.21, "foreign_cores_max": 1.027, "kernel_thread_cores_max": 0.108, "agent_processes_max": 3, "agent_tools": ["bash"], "agent_cores_max": 0.163, "foreign_builds": [], "top_foreign": [".claude-wrapped", "MainThread", "nsncd"], "control_fps": 30295.267718284824, "control_loss_pct": 0, "profiled_fps": 30940, "profiled_loss_pct": 0, "mutex_ms": 38.819, "idle_workers": 0, "groupstate_us": 709, "groupstate_share_pct": 1.83, "by_lock_us": {"lock<kio::State<moq_net::model::track::TrackState>>": 37906, "lock<kio::State<moq_net::model::group::GroupState>>": 709, "lock<kio::State<moq_net::model::resume::Route>>": 67}, "groupstate_stacks": {"<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_read_frame;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#24}::poll_read_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall": 709}}
{"run": "r1A", "label": "current", "workers": 1, "window": [1793732638.6, 1793732665.0], "samples": 15, "load1_max": 11.83, "load1_at_start": 11.83, "busy_cores_max": 7.698, "foreign_cores_mean": 0.032, "foreign_cores_max": 0.099, "kernel_thread_cores_max": 0.02, "agent_processes_max": 3, "agent_tools": ["bash"], "agent_cores_max": 0.015, "foreign_builds": [], "top_foreign": ["nsncd"], "control_fps": 20632.926585317065, "control_loss_pct": 34.5475910693302, "profiled_fps": 21438, "profiled_loss_pct": 32.49903735078937, "mutex_ms": 0.0, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": null, "by_lock_us": {}, "groupstate_stacks": {}}
{"run": "r1A", "label": "current", "workers": 4, "window": [1793732665.0, 1793732727.4], "samples": 32, "load1_max": 9.07, "load1_at_start": 8.22, "busy_cores_max": 6.635, "foreign_cores_mean": 0.054, "foreign_cores_max": 0.707, "kernel_thread_cores_max": 0.236, "agent_processes_max": 3, "agent_tools": ["bash"], "agent_cores_max": 0.02, "foreign_builds": [], "top_foreign": ["MainThread", "nsncd"], "control_fps": 31239.6, "control_loss_pct": 0, "profiled_fps": 31665.4, "profiled_loss_pct": 0, "mutex_ms": 3.489, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": 0.0, "by_lock_us": {"lock<moq_net::model::track::TrackState>": 3322, "lock<kio::channel::Waiters>": 57, "lock<moq_net::model::resume::Route>": 16, "lock<moq_net::model::expiry::Parked>": 13}, "groupstate_stacks": {}}
{"run": "r1A", "label": "current", "workers": 16, "window": [1793732727.4, 1793732786.9], "samples": 31, "load1_max": 7.99, "load1_at_start": 7.07, "busy_cores_max": 7.542, "foreign_cores_mean": 0.046, "foreign_cores_max": 0.272, "kernel_thread_cores_max": 0.158, "agent_processes_max": 4, "agent_tools": ["bash"], "agent_cores_max": 0.025, "foreign_builds": [], "top_foreign": [".claude-wrapped", "MainThread", "nsncd"], "control_fps": 31216.52991795077, "control_loss_pct": 0, "refused": true, "refusal": ["profile: worker moq-quic-7 (2953414) has nonalternating switch records"]}

r1A, base (pre-epoch), 16 workers: full mutex.folded

<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 21923
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 3839
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1324
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 1194
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 968
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 894
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 808
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 759
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_read_frame;poll<moq_net::model::group::GroupState, moq_net::model::group::{impl#24}::poll_read_frame::{closure_env#1}, core::result::Result<core::option::Option<moq_net::model::group::Inflate>, moq_net::error::Error>>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 709
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;next_filled;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 699
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 512
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 497
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 497
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 494
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 464
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 372
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 311
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;poll<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_datagram::{closure_env#0}, core::option::Option<(moq_net::model::datagram::Stored, usize)>>, core::result::Result<core::option::Option<(moq_net::model::datagram::Stored, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 190
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;<moq_net::model::track::Dynamic>::poll_requested_fill;poll_requested;poll_ref<moq_net::model::track::TrackState, moq_net::model::track::fill::poll_requested::{closure_env#1}, moq_net::error::Error>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 185
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 155
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 155
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 139
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 139
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 139
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 133
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 129
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 117
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;is_live;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 114
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 112
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#23}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 112
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;poll_closed;<moq_net::model::track::TrackState>::poll_closed;<kio::weak::ConsumerWeak<moq_net::model::track::TrackState>>::poll_closed;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 93
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 68
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;read<moq_net::model::resume::Route>;lock<kio::State<moq_net::model::resume::Route>>;lock<kio::State<moq_net::model::resume::Route>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 67
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;poll_recv_wire_datagram;poll_recv_stored_datagram;<moq_net::model::track::Cursor>::poll_recv_datagram;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 62
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;<moq_net::model::track::Consumer>::serving;with_state<core::option::Option<moq_net::model::resume::Consumer>, moq_net::model::track::{impl#22}::serving::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 50
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 41
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_used;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 40
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 35
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 27
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;poll<moq_net::model::track::{impl#35}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_recv_with::{closure_env#1}, core::option::Option<(moq_net::model::group::Handle, usize)>>, core::result::Result<core::option::Option<(moq_net::model::group::Handle, usize)>, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 25
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 21
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 19
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::poll_drift;poll<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_drift::{closure_env#1}, moq_net::model::track::Drift>, core::result::Result<moq_net::model::track::Drift, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 19
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::wanted;is_used;is_closed<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 14
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#23}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 10
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;poll_recv_with;poll_stale;poll<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#35}::poll_stale::{closure_env#0}, bool>, core::result::Result<bool, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8

(Written by Claude Opus 5.5)

@kixelated

kixelated commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reduced profiling evidence, nixos2 repeat, part 2/2. The pre-epoch 4-worker fold from r1A, and the void runs. A1 (run.C3e0Dz, pre-epoch) overlapped another agent's cargo builds and refused its base 16-worker capture for nonalternating switch records. r2B (run.3mg0gy, epoch d854bf98b1) ran beside another session's moq bench builds, which used 5 to 6 cores throughout; perf dropped an event at its base 16-worker capture. Neither enters the results; their rows are retained for the record. A1's windows are recomputed from the retained files without perf.data, so they are slightly narrower.

r1A, base (pre-epoch), 4 workers: full mutex.folded

<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 172
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 63
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;poll_expired;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 60
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_next_frame;poll_resumed<moq_net::model::frame::Consumer, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Consumer>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_next_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Consumer>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 49
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 48
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 41
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};<moq_net::model::track::GroupExpiry as moq_net::model::group::Expiry>::is_expired;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 35
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 32
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 30
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 26
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;poll_peek_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 22
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 21
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 20
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 20
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;<moq_net::model::resume::Reader>::let_go;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 16
<moq_net::lite::subscriber::TrackServeRun as kio::task::Task>::poll;<moq_net::lite::subscriber::ServeLoop>::poll;poll_unwanted;<moq_net::model::track::Demand>::poll_wanted_changed;poll_state;<kio::weak::ProducerWeak<moq_net::model::track::TrackState>>::poll_used;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 15
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_stored_datagram;<moq_net::model::resume::Subscriber>::poll_recv_datagram;poll_recv_datagram;poll_datagram_once;<moq_net::model::resume::Reader>::let_go;and_then<&moq_net::model::resume::Copy, u64, moq_net::model::resume::{impl#5}::let_go::{closure_env#1}>;{closure#1};<moq_net::model::track::Consumer>::latest_sequence;with_state<core::option::Option<u64>, moq_net::model::track::{impl#22}::latest_sequence::{closure_env#0}>;{closure#0};read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 13
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::resume::Reader>::poll_lowest;refused;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 13
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;<moq_net::model::track::Subscriber>::poll_finished;poll_finished;<moq_net::model::resume::Reader>::poll_finished;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 12
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_open;poll_given_up;<moq_net::model::group::Consumer>::poll_finished;poll_resumed<u64, moq_net::model::group::{impl#23}::poll_finished::{closure_env#0}>;{closure#0};<moq_net::model::group::Consumer>::poll_finished_once;poll_expired_if_blocked<u64>;<moq_net::model::group::Consumer>::poll_expiry;is_some_and<&alloc::sync::Arc<dyn moq_net::model::group::Expiry, alloc::alloc::Global>, moq_net::model::group::{impl#23}::poll_expiry::{closure_env#0}>;{closure#0};and_then<&alloc::boxed::Box<moq_net::model::resume::Recover, alloc::alloc::Global>, core::time::Duration, moq_net::model::group::{impl#23}::poll_expiry::{closure#0}::{closure_env#0}>;{closure#0};<moq_net::model::resume::Recover>::poll_budget;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 12
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;is_live;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 11
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll_drop::{closure_env#0},;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 11
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::resume::Reader>::poll_serving;<moq_net::model::resume::Reader>::sync;<moq_net::model::resume::Reader>::learn_end;poll_finished;<moq_net::model::track::Cursor>::poll::<<moq_net::model::track::Subscriber>::poll_finished::{closure#0}, moq_net::model::subscription::Position>;poll<moq_net::model::track::TrackState, moq_net::model::track::{impl#35}::poll::{closure_env#0}<moq_net::model::track::{impl#38}::poll_finished::{closure_env#0}, moq_net::model::subscription::Position>, core::result::Result<moq_net::model::subscription::Position, moq_net::error::Error>>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 11
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<kio::task::Tasks<moq_net::lite::publisher::GroupServe>>::poll;poll;poll_serve;<moq_net::model::group::Consumer>::poll_read_objects::<8>;poll_read_object;<moq_net::model::group::Consumer>::poll_read_frame;poll_resumed<moq_net::model::frame::Frame, fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>>;call_mut<fn(&mut moq_net::model::group::Consumer, &kio::waiter::Waiter) -> core::task::poll::Poll<core::result::Result<core::option::Option<moq_net::model::frame::Frame>, moq_net::error::Error>>,;<moq_net::model::group::Consumer>::poll_read_frame_once;poll_expired_if_blocked<core::option::Option<moq_net::model::frame::Frame>>;<moq_net::model::group::Consumer>::poll_expiry;expiry_pending;expiry_pending;read<moq_net::model::group::GroupState>;lock<kio::State<moq_net::model::group::GroupState>>;lock<kio::State<moq_net::model::group::GroupState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 9
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;<moq_net::lite::publisher::TrackRun>::poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::probe;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 9
<moq_net::lite::publisher::Control>::poll_serve;poll<moq_net::lite::publisher::SubscribeServe>;poll_serve<moq_net::lite::publisher::SubscribeServe>;<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_forward_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;poll_drop;<moq_net::model::track::Subscriber>::poll_drop;poll_drop;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::session::Driver>::poll_protocol;poll;<moq_net::lite::subscriber::UniAccept>::poll;<kio::task::Tasks<moq_net::lite::subscriber::UniServe>>::poll;poll;<moq_net::lite::subscriber::UniServe>::poll_serve;poll_serve;<moq_net::model::track::Producer>::subscription;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8
<moq_net::lite::publisher::SubscribeServe as moq_net::lite::publisher::Request>::poll;poll_step;poll_recv_next;poll_recv_group;<moq_net::model::track::Subscriber>::poll_group;poll_group;poll_group;poll_group_once;<moq_net::model::track::Subscriber>::poll_group;<moq_net::model::track::Cursor>::poll_recv_group;<moq_net::model::track::Cursor>::sync;read<moq_net::model::track::TrackState>;lock<kio::State<moq_net::model::track::TrackState>>;lock<kio::State<moq_net::model::track::TrackState>>;lock;<std::sys::sync::mutex::futex::Mutex>::lock_contended;std::sys::pal::unix::futex::futex_wait;syscall 8

Void runs, reduced

{"run": "A1", "label": "base", "workers": 1, "window": [1793731020.0, 1793731051.0], "samples": 18, "load1_max": 4.01, "load1_at_start": 1.86, "busy_cores_max": 7.189, "foreign_cores_mean": 0.157, "foreign_cores_max": 1.051, "kernel_thread_cores_max": 0.03, "agent_processes_max": 2, "agent_tools": ["bash", "just", "quest", "tail"], "agent_cores_max": 0.02, "foreign_builds": ["cargo", "gcc", "rustc"], "top_foreign": [".localsearch-3-", "build-script-ma", "cc1", "clippy-driver", "nsncd"], "control_fps": 22442.56831815152, "control_loss_pct": 48.66332497911445, "profiled_fps": 21852.2, "profiled_loss_pct": 49.15603128859613, "mutex_ms": 0.0, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": null, "by_lock_us": {}}
{"run": "A1", "label": "base", "workers": 4, "window": [1793731051.0, 1793731131.0], "samples": 41, "load1_max": 10.2, "load1_at_start": 4.01, "busy_cores_max": 9.833, "foreign_cores_mean": 0.668, "foreign_cores_max": 2.447, "kernel_thread_cores_max": 0.088, "agent_processes_max": 2, "agent_tools": ["bash", "just", "quest", "tail"], "agent_cores_max": 0.074, "foreign_builds": ["cargo", "gcc", "ld.lld", "rustc"], "top_foreign": [".claude-wrapped", "MainThread", "build-script-ma", "cargo", "cc1plus", "clippy-driver", "nsncd", "rustc"], "control_fps": 30712.286158631414, "control_loss_pct": 0, "profiled_fps": 31535, "profiled_loss_pct": 0, "mutex_ms": 17.085, "idle_workers": 0, "groupstate_us": 20, "groupstate_share_pct": 0.12, "by_lock_us": {"lock<kio::State<moq_net::model::track::TrackState>>": 16802, "lock<kio::State<moq_net::model::group::GroupState>>": 20, "lock<kio::State<moq_net::model::resume::Route>>": 13}}
{"run": "A1", "label": "base", "workers": 16, "window": [1793731131.0, 1793731216.0], "samples": 44, "load1_max": 18.49, "load1_at_start": 10.2, "busy_cores_max": 11.181, "foreign_cores_mean": 0.986, "foreign_cores_max": 2.633, "kernel_thread_cores_max": 0.156, "agent_processes_max": 3, "agent_tools": ["bash", "bwrap", "codex-code-mode", "just", "quest", "tail"], "agent_cores_max": 0.521, "foreign_builds": ["cargo", "gcc", "ld.lld", "rustc"], "top_foreign": [".claude-wrapped", "MainThread", "cargo", "codex", "nsncd", "rustc"], "control_fps": 30715.6, "control_loss_pct": 0, "refused": true, "refusal": ["profile: worker moq-quic-7 (2895759) has nonalternating switch records"]}
{"run": "r2B", "label": "base", "workers": 1, "window": [1793733590.4, 1793733625.4], "samples": 19, "load1_max": 8.46, "load1_at_start": 7.86, "busy_cores_max": 9.784, "foreign_cores_mean": 5.061, "foreign_cores_max": 5.982, "kernel_thread_cores_max": 0.212, "agent_processes_max": 2, "agent_tools": [], "agent_cores_max": 0.02, "foreign_builds": ["cargo", "gcc", "ld.lld", "rustc"], "top_foreign": ["nsncd", "rustc"], "control_fps": 11478, "control_loss_pct": 82.25338491295938, "profiled_fps": 12456.4, "profiled_loss_pct": 77.47536367902393, "mutex_ms": 0.0, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": null, "by_lock_us": {}}
{"run": "r2B", "label": "base", "workers": 4, "window": [1793733625.4, 1793733716.3], "samples": 46, "load1_max": 15.36, "load1_at_start": 8.46, "busy_cores_max": 11.952, "foreign_cores_mean": 5.721, "foreign_cores_max": 6.344, "kernel_thread_cores_max": 1.044, "agent_processes_max": 3, "agent_tools": [], "agent_cores_max": 0.067, "foreign_builds": ["cargo", "gcc", "ld.lld", "rustc"], "top_foreign": ["rustc"], "control_fps": 31373.2, "control_loss_pct": 0, "profiled_fps": 31510.604241696677, "profiled_loss_pct": 0, "mutex_ms": 1.747, "idle_workers": 0, "groupstate_us": 0, "groupstate_share_pct": 0.0, "by_lock_us": {"lock<kio::State<moq_net::model::track::TrackState>>": 1737}}
{"run": "r2B", "label": "base", "workers": 16, "window": [1793733716.3, 1793733792.6], "samples": 38, "load1_max": 23.15, "load1_at_start": 15.36, "busy_cores_max": 11.971, "foreign_cores_mean": 5.826, "foreign_cores_max": 6.342, "kernel_thread_cores_max": 1.027, "agent_processes_max": 2, "agent_tools": [], "agent_cores_max": 0.305, "foreign_builds": ["cargo", "rustc"], "top_foreign": ["rustc"], "control_fps": 29904.687847145076, "control_loss_pct": 5.644573925710124, "refused": true, "refusal": ["profile: perf dropped events (PERF_RECORD_LOST lost 1); the recording is incomplete"]}

Run log (epoch start, end, exit)

A1	d854bf98b1~1	1793730735.363673638	1793731227.604980683	1
r1A	d854bf98b1~1	1793732118.995106636	1793732787.134877555	1
r2B	d854bf98b1	1793733196.534474765	1793733792.967717251	1

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 3, 2026 12:02
@kixelated
Merge remote-tracking branch 'origin/main' into HEAD
27c773e
@kixelated @claude
docs(quest): accept the log lock profile and delete its quest
37d59f3 
The maintainer accepted no lock-contention regression attributable to
#6782: the pre-epoch run shows the same GroupState wait stack at a similar
wait (709 vs 599 us). Record that in the unbounded-groups epic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated mentioned this pull request Nov 3, 2026
Merged
@kixelated kixelated changed the title docs(quest): record mixed log lock profile evidence docs(quest): accept the log lock profile and delete its quest Nov 3, 2026
@kixelated
kixelated marked this pull request as ready for review November 3, 2026 20:06
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9e84e30c-3dbe-4159-a60e-1ae9245b969d
Commits

Reviewing files that changed from the base of the PR and between 151ed38 and 686ba58.

Files selected for processing: 2
  • quest/m1/unbounded-groups/README.md
  • quest/m1/unbounded-groups/log-lock-profile.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

kixelated
kixelated commented Nov 3, 2026
View reviewed changes

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 37d59f3.

[P2] Qualify the cross-host wait figures in the durable conclusion (quest/m1/unbounded-groups/README.md:117–120). The 709 µs pre-epoch wait was measured on nixos2, while the 599 µs epoch wait was on desktop. The PR's own evidence says those hosts' totals are not comparable, the quiet same-host repeat was not completed, and the original share goal has no valid measurement. Placing the two values side by side without that qualification can make the retained epic read like a controlled before/after result. Keep the maintainer's accepted decision, but state alongside it that the figures are cross-host and the comparison was inconclusive.

Static GitHub-only review. I inspected the recorded evidence and did not rerun profiles or independently verify the host activity.

pullfrog[bot]
pullfrog Bot reviewed Nov 3, 2026
View reviewed changes

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The added durable conclusion pairs two host-specific wait figures as a controlled before/after, but the PR body itself says those hosts' measurements are not comparable. Qualify the figures before this lands in the epic.

Reviewed changes

  • Delete the log-lock quest — removes quest/m1/unbounded-groups/log-lock-profile.md, whose goal was a just bench-profile lock video-heavy before/after #6782 comparison.
  • Drop the epic link — removes the quest's line from the "Required" list in quest/m1/unbounded-groups/README.md.
  • Record the accepted conclusion — adds four lines to the epic's storage decision stating no lock-contention regression is attributable to #6782, citing #7742.

No dangling references remain: no other tracked file mentions the deleted quest or its link, and the net diff is only these two files.

Pullfrog  ｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) ｜ 𝕏

Comment thread quest/m1/unbounded-groups/README.md Resolved
kixelated added a commit that referenced this pull request Nov 3, 2026
@kixelated @claude
quest: drop the bench lock's link to the log lock profile
7ef3984 
#7742 deletes quest/m1/unbounded-groups/log-lock-profile.md, so the
link would dangle once it lands; #7742 stays cited as the evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated @claude
docs(quest): name the host beside each log lock wait figure
686ba58 
The 709 and 599 us waits come from different hosts, so the epic now says
they are not comparable while keeping the accepted conclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
kixelated commented Nov 3, 2026
View reviewed changes

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 686ba58.

The prior cross-host qualification issue is addressed: the retained epic now names each host and says the figures are not comparable and no quiet same-host repeat was completed. No additional actionable findings in this update.

Verification: static GitHub-only inspection; I did not rerun profiles or independently verify host activity.

pullfrog[bot]
pullfrog Bot approved these changes Nov 3, 2026
View reviewed changes

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

This run re-reviewed the single commit since the prior pullfrog review (686ba58), which qualifies the durable conclusion added by the quest deletion.

  • Qualified the cross-host wait figures — the epic's storage decision now names nixos2 beside the 709 µs pre-epoch wait and desktop beside the 599 µs epoch wait, and states the figures are not comparable with no quiet same-host repeat completed.

The prior concern (cross-host figures reading as a controlled before/after) is addressed, and no dangling references to the deleted quest remain.

Pullfrog  ｜ View workflow run ｜ Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) ｜ 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit de94d96 Nov 3, 2026
9 of 10 checks passed
@kixelated
kixelated deleted the quest/m1/unbounded-groups/log-lock-profile branch November 3, 2026 22:09
kixelated added a commit that referenced this pull request Nov 4, 2026
@kixelated @claude
Merge origin/main into quest-plan/2026-11-03-followups
8d96c44 
Resolve the modify/delete conflicts on quests main finished
(m0/ci/shared-target.md in #7715, m1/resume-model-7542.md in #7705), and
drop or repoint links and Required edges to quests main deleted:
m1/ietf-resume-behind.md (#7708), m1/subscription-delay.md (#7697),
m1/unbounded-groups/log-lock-profile.md (#7742) and m1/moq-crypto.md
(#7702).

#7705 did not replay #7739's seed 17893651, which this PR had added to
the resume-model quest, so it moves to m1/resume-seed-17893651.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated mentioned this pull request Nov 4, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pullfrog pullfrog[bot] pullfrog[bot] approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kixelated