From d94cceef60f0127fcd2576d229a18cb18777eb43 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:33:37 -0800 Subject: [PATCH 1/7] chore(quest): claim quest/m0/test-flakes-2/gst-reconnect-interleave.md Quest-Claim: 5d978dbd13fad4c6 From 9fc50fdb558a615937b677bd83ee95f03b23c0ac Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:51:44 -0800 Subject: [PATCH 2/7] docs(quest): record reconnect flake reproduction evidence Co-Authored-By: Codex --- .../test-flakes-2/gst-reconnect-interleave.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index ee5b8b42e36..120e9d55e87 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -11,6 +11,23 @@ each failure below and shows the loaded run that reproduced it now passing. ## Plan +Investigation 2026-11-04: the unchanged ignored test passed once alone, 50 +repetitions alongside `just check`, and 100 repetitions alongside +`just check --all`, all through `quest gate` in the pinned Nix shell. +The first accompanying check passed. Cargo build locks serialized parts +of the full-workspace sweep, so these runs do not prove the race absent. +No failure was reproduced and no source change is justified yet; keep the +ignore until a failing schedule is captured. + +The historical `Unroutable` in +[moq-dev/moq#7296](https://github.com/moq-dev/moq/pull/7296) came from +`stop()` unwrapping the session result (line 3822 at merged commit +`103e98e3624c193fc320e6513d0fb473a6cbe381`), not from +`publication.announce()`. The failing session operation remains unknown. +Media delivery still does not acknowledge catalog reconciliation, so a +controlled schedule around relist reconciliation and route withdrawal +is the next investigative lead, not a verified fix. + Planned 2026-11-04 with the maintainer, from [moq-dev/moq#7888](https://github.com/moq-dev/moq/pull/7888)'s suggested follow-ups. No quest or open PR tracked it. From 1f7adfb384411ef8edea66ef2083e4ff02ac711d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:51:07 -0800 Subject: [PATCH 3/7] docs(quest): wait for gst reconnect failure evidence Co-Authored-By: GPT-6.1 Sol --- .../test-flakes-2/gst-reconnect-interleave.md | 41 ++++++++++++++----- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index 120e9d55e87..f6d51f06ed9 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -11,22 +11,30 @@ each failure below and shows the loaded run that reproduced it now passing. ## Plan -Investigation 2026-11-04: the unchanged ignored test passed once alone, 50 +Investigation in [moq-dev/moq#7932](https://github.com/moq-dev/moq/pull/7932), +2026-11-04: the unchanged ignored test passed once alone, 50 repetitions alongside `just check`, and 100 repetitions alongside `just check --all`, all through `quest gate` in the pinned Nix shell. The first accompanying check passed. Cargo build locks serialized parts of the full-workspace sweep, so these runs do not prove the race absent. -No failure was reproduced and no source change is justified yet; keep the -ignore until a failing schedule is captured. +All 151 executions passed. No failure was reproduced; keep the ignore +until the cause is verified and fixed. The historical `Unroutable` in [moq-dev/moq#7296](https://github.com/moq-dev/moq/pull/7296) came from `stop()` unwrapping the session result (line 3822 at merged commit -`103e98e3624c193fc320e6513d0fb473a6cbe381`), not from +[`103e98e`](https://github.com/moq-dev/moq/commit/103e98e3624c193fc320e6513d0fb473a6cbe381)), not from `publication.announce()`. The failing session operation remains unknown. Media delivery still does not acknowledge catalog reconciliation, so a controlled schedule around relist reconciliation and route withdrawal -is the next investigative lead, not a verified fix. +remains an unverified lead. + +Decided 2026-11-05 (maintainer, /quest-plan): capture a fresh failing +loaded or merge-queue run with the session error attached first (option 2). +Do not add scheduling instrumentation now. The evidence-only PR can land; +the external wait belongs in this quest. Keep a code change only if it +makes the next failure report its session error; otherwise leave this as +a quest-only change. Planned 2026-11-04 with the maintainer, from [moq-dev/moq#7888](https://github.com/moq-dev/moq/pull/7888)'s suggested @@ -76,14 +84,25 @@ in a fast run. Find each from the code under test before changing the test. moqsrc bug, fixed in moqsrc ([moq-dev/moq#7368](https://github.com/moq-dev/moq/pull/7368) made the actor own the pad). -- Find which call returned `Unroutable` in #7296's run, and whether +- Find which call returned `Unroutable` in the historical run, and whether `publication.announce` or a follow can meet an unroutable path in the window after `unannounce`. -Reproduce first: run moq-gst's tests together with the rest of the -workspace under CPU contention, repeating this test in a loop beside a -loaded `just check`. If it doesn't reproduce in a reasonable number -of runs, report that and the runs tried rather than changing the test -blind. +After the capture below, identify the failing session operation and use +the recorded conditions to reproduce and fix each failure at its cause. +Keep the test ignored until the loaded reproduction passes with the fix. Public API: none. Wire: none. + +## Waiting + +- moq-dev/moq: a fresh failing loaded or merge-queue run of + `restarts_under_reconnects_interleave_on_one_pad` with the session error + attached. Check subsequent Actions/merge-queue logs and PR failure reports; + because the test is ignored, advance the capture by explicitly running it + with its fully qualified name and `--ignored --exact` under load through + `quest gate` in the pinned + Nix shell. Attach the run or log, tested commit, load conditions, and full + session error chain to this quest. If the failure output lacks the session + error, surface that error before collecting another run. Clear this wait + when that evidence is attached, then resume root-cause investigation. From 57573bf9a4371f44915153bc616f86b1ce1d0e73 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:06:02 -0800 Subject: [PATCH 4/7] docs(quest): remove stale announcement investigation lead Co-Authored-By: GPT-6.1 Sol --- quest/m0/test-flakes-2/gst-reconnect-interleave.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index f6d51f06ed9..56f2fa9f511 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -84,9 +84,6 @@ in a fast run. Find each from the code under test before changing the test. moqsrc bug, fixed in moqsrc ([moq-dev/moq#7368](https://github.com/moq-dev/moq/pull/7368) made the actor own the pad). -- Find which call returned `Unroutable` in the historical run, and whether - `publication.announce` or a follow can meet an unroutable path in the - window after `unannounce`. After the capture below, identify the failing session operation and use the recorded conditions to reproduce and fix each failure at its cause. From 43010d64244fade7fc9cef8edc5e54bb2210b5a7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:06:10 -0800 Subject: [PATCH 5/7] quest: name the load and command for the gst interleave capture Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/gst-reconnect-interleave.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index 56f2fa9f511..28d067e7bbd 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -96,10 +96,13 @@ Public API: none. Wire: none. - moq-dev/moq: a fresh failing loaded or merge-queue run of `restarts_under_reconnects_interleave_on_one_pad` with the session error attached. Check subsequent Actions/merge-queue logs and PR failure reports; - because the test is ignored, advance the capture by explicitly running it - with its fully qualified name and `--ignored --exact` under load through - `quest gate` in the pinned - Nix shell. Attach the run or log, tested commit, load conditions, and full + because the test is ignored and has no CI caller + (`sh/gh/test-inventory.allow`), no Actions log will show it; advance the + capture by hand. `quest gate` waits for host load to drop, so supply the + load inside the gated run: in the pinned Nix shell, run + `quest gate sh -c 'just check --all & for i in $(seq 100); do cargo test -p moq-gst --lib -- source::imp::tests::restarts_under_reconnects_interleave_on_one_pad --ignored --exact || break; done; wait'` + so the repetitions loop beside a loaded `just check --all`, as in the Plan. + Attach the run or log, tested commit, load conditions, and full session error chain to this quest. If the failure output lacks the session error, surface that error before collecting another run. Clear this wait when that evidence is attached, then resume root-cause investigation. From 60d03dee91c6f5cfafd9aa0227f89f564e09927a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:07:21 -0800 Subject: [PATCH 6/7] quest: target session_tests in the capture filter Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/gst-reconnect-interleave.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index 28d067e7bbd..9090b10b232 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -100,7 +100,7 @@ Public API: none. Wire: none. (`sh/gh/test-inventory.allow`), no Actions log will show it; advance the capture by hand. `quest gate` waits for host load to drop, so supply the load inside the gated run: in the pinned Nix shell, run - `quest gate sh -c 'just check --all & for i in $(seq 100); do cargo test -p moq-gst --lib -- source::imp::tests::restarts_under_reconnects_interleave_on_one_pad --ignored --exact || break; done; wait'` + `quest gate sh -c 'just check --all & for i in $(seq 100); do cargo test -p moq-gst --lib -- source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad --ignored --exact || break; done; wait'` so the repetitions loop beside a loaded `just check --all`, as in the Plan. Attach the run or log, tested commit, load conditions, and full session error chain to this quest. If the failure output lacks the session From efedb9609b4966b0756da83634c661ad7e055fd2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:57:48 -0800 Subject: [PATCH 7/7] quest(gst-reconnect): capture command keeps both exit statuses and stops with the load Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/gst-reconnect-interleave.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/gst-reconnect-interleave.md b/quest/m0/test-flakes-2/gst-reconnect-interleave.md index 9090b10b232..1bd9ace0a1c 100644 --- a/quest/m0/test-flakes-2/gst-reconnect-interleave.md +++ b/quest/m0/test-flakes-2/gst-reconnect-interleave.md @@ -100,8 +100,9 @@ Public API: none. Wire: none. (`sh/gh/test-inventory.allow`), no Actions log will show it; advance the capture by hand. `quest gate` waits for host load to drop, so supply the load inside the gated run: in the pinned Nix shell, run - `quest gate sh -c 'just check --all & for i in $(seq 100); do cargo test -p moq-gst --lib -- source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad --ignored --exact || break; done; wait'` - so the repetitions loop beside a loaded `just check --all`, as in the Plan. + `quest gate sh -c 'just check --all & c=$!; s=0; for i in $(seq 100); do kill -0 $c 2>/dev/null || break; cargo test -p moq-gst --lib -- source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad --ignored --exact || { s=1; break; }; done; wait $c || s=1; exit $s'` + so the repetitions loop only while `just check --all` still loads the host, as + in the Plan, and the command fails if either a repetition or the check fails. Attach the run or log, tested commit, load conditions, and full session error chain to this quest. If the failure output lacks the session error, surface that error before collecting another run. Clear this wait