diff --git a/artifacts/findings.yaml b/artifacts/findings.yaml index 00f6a57..0204a81 100644 --- a/artifacts/findings.yaml +++ b/artifacts/findings.yaml @@ -9417,13 +9417,13 @@ artifacts: - id: AFD-150 type: ai-found-defect title: "The Renode smoke job's Mono negative control is nondeterministic: NC1 expected exit 3, got rc=1, then an exit 134 abort" - status: open - description: "Main push run 40664674271 (b810466, #304) failed 'Renode RT1176 smoke' with 'NC0 the real archive -> PASS', then 'NC1 expected an exit-3 failure, got rc=1' and 'FAIL: unexpected exit 134: SMOKE FAIL: boot line not seen within 2 s'. The same commit's PR run passed, and every later main run passed both runtimes. 134 is SIGABRT. The failure path is the boot-failure exit (Environment.Exit(3)) on the legacy Mono runtime, where rt1176-smoke-check.resc already documents that exit codes are unreliable on load errors. DENOMINATOR: 1 run with this signature of the 12 CI runs since 2026-10-28 whose logs could be read for it; 3 other Renode failures in that window were PR iterations on #299 whose logs no longer carry a signature. With the job now REQUIRED (AFD-149), a recurrence blocks a merge until re-run. Not yet diagnosed; not yet reproduced locally (local Renode is the .NET build)." + status: resolved + description: "Main push run 40664674271 (b810466, #304) failed 'Renode RT1176 smoke' with 'NC0 the real archive -> PASS', then 'NC1 expected an exit-3 failure, got rc=1' and 'FAIL: unexpected exit 134: SMOKE FAIL: boot line not seen within 2 s'. The same commit's PR run passed, and every later main run passed both runtimes. 134 is SIGABRT. The failure path is the boot-failure exit (Environment.Exit(3)) on the legacy Mono runtime, where rt1176-smoke-check.resc already documents that exit codes are unreliable on load errors. DENOMINATOR: 1 run with this signature of the 12 CI runs since 2026-10-28 whose logs could be read for it; 3 other Renode failures in that window were PR iterations on #299 whose logs no longer carry a signature. With the job now REQUIRED (AFD-149), a recurrence blocks a merge until re-run. Not yet diagnosed; not yet reproduced locally (local Renode is the .NET build). DIAGNOSED AND FIXED 2026-11-03: the failure is not in the boot assertion but in NC1's demand for the literal exit code. In both CI failures verify.sh rejected the mutated archive (rc=1) and the .resc printed 'SMOKE FAIL: boot line not seen within 2 s' before Mono aborted in Environment.Exit(3) with 134 (SIGABRT); NC1 grepped for 'exit 3' and so reported the control as broken although the assertion had bitten. Same Mono exit-channel unreliability the .resc documents for load errors and NC2 already allows for. FIX (tools/renode-model/verify.sh): NC1 accepts, ONLY for a runtime detected as Mono, rejection AND the FAIL verdict line AND no PASS line with exit 134; .NET keeps exact exit 3. Measured: real .NET Renode 1.16.1 self-test NC0-NC5 pass via the exit-3 path; a stand-in Mono Renode that prints the verdict then SIGABRTs passes via the new branch; one-variable controls stay red: the abort WITHOUT the verdict line (runtime mono) and the same abort from a runtime detected as dotnet. Not reproduced on real Mono locally (local Renode is the .NET build); the CI Mono runtime is the confirmation." fields: detected-by: the AFD-149 audit of main's push runs discovered-at: 2026-10-30 severity: minor - triage-status: open + triage-status: resolved provenance: created-by: ai model: claude-opus-5-5 diff --git a/tools/renode-model/verify.sh b/tools/renode-model/verify.sh index 3172ba8..6c3b4a5 100755 --- a/tools/renode-model/verify.sh +++ b/tools/renode-model/verify.sh @@ -90,7 +90,23 @@ if [ "${1:-}" = "--self-test" ]; then ctl() { rc=0; RENODE="$RENODE" "$me" "$1" > "$tmp/o" 2>&1 || rc=$?; echo $rc; } r=$(ctl "$good"); [ "$r" = 0 ] && echo "NC0 the real archive -> PASS (exit 0) [positive control]" || { echo "NC0 expected 0 got $r"; cat "$tmp/o"; exit 1; } mutate "$tmp/nc1.tgz" "p=d+'/rt1176-smoke-check.resc'; s=open(p).read().replace('JESS-RT1176 boot OK\" \"','JESS-RT1176 NEVER PRINTED\" \"'); open(p,'w').write(s)" - r=$(ctl "$tmp/nc1.tgz"); [ "$r" = 1 ] && grep -q 'exit 3' "$tmp/o" && echo "NC1 check asserts a line never printed -> FAIL, exit 3 [the assertion inside the .resc bites]" || { echo "NC1 expected an exit-3 failure, got rc=$r"; cat "$tmp/o"; exit 1; } + # NC1 — the boot assertion bites. On .NET the .resc exits 3, exactly. On MONO the same + # Environment.Exit(3) sometimes ABORTS (exit 134, SIGABRT) AFTER the verdict line is printed: + # measured twice in CI (AFD-150: 2 of 13 runs), each time with 'SMOKE FAIL: boot line not seen' + # in the output and verify.sh rejecting (rc=1). The assertion bit; the exit channel is what + # Mono garbles, exactly as the .resc documents for load errors. So, as in NC2, the Mono allowance + # is granted only to a DETECTED Mono, and it still requires the rejection AND the FAIL line AND + # no PASS line — it accepts 3 or 134 as the code, nothing else. .NET keeps the exact exit 3. + rt=$(runtime_of "$RENODE") + r=$(ctl "$tmp/nc1.tgz") + if [ "$r" = 1 ] && grep -q 'exit 3' "$tmp/o"; then + echo "NC1 check asserts a line never printed -> FAIL, exit 3 [the assertion inside the .resc bites]" + elif [ "$r" = 1 ] && [ "$rt" = mono ] && grep -q 'unexpected exit 134' "$tmp/o" \ + && grep -q 'SMOKE FAIL: boot line not seen' "$tmp/o" && ! grep -q "$PASS_LINE" "$tmp/o"; then + echo "NC1 check asserts a line never printed -> FAIL, verdict printed then exit 134 [mono aborted in Exit(3) — AFD-150; the assertion still bit]" + else + echo "NC1 expected the boot assertion to bite (exit 3, or on Mono the FAIL line then exit 134); runtime=$rt rc=$r"; cat "$tmp/o"; exit 1 + fi mutate "$tmp/nc2.tgz" "p=d+'/rt1176-smoke-check.resc'; s=open(p).read().replace('\$ORIGIN/','@hardware/renode/'); open(p,'w').write(s)" # NC2 — a LOAD error. On EVERY runtime verify.sh must REJECT it (rc=1). On .NET the .resc # itself exits 4. On Mono it exits 0 without the PASS line (the documented limitation), and