Repository navigation
ci: a hung test module fails inside its pass under a session timeout (#447) - #449
Merged
Merged
Conversation
…447) Each test pass runs with a 25-minute --timeout, about four times the slowest measured module, and a failed main pass also writes its test listing, so the rows still in flight are among those listed but not recorded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every CI test pass now runs under a 25-minute session
--timeout, so a hung test module fails inside its step instead of at the job timeout. A failed main pass also writes its test listing, which narrows the rows that were still in flight. For #447.Why
In #442's run 39031358513, the Linux
OpenCode.Sdk.Testsnet11.0 module hung for about 50 minutes. Without a TRX, nothing named the row, and it did not reproduce in four configurations on dev-01 (#447).What it does, and what it does not
What it does. The guard fails inside the step and narrows the search.
What it does not do. It does not name the row in flight. Measured locally with
--timeout 8sonOpenCode.Sdk.Tools.Tests:RunInfosaying the host was terminated because the session was aborted.Hence the listing. When the main pass fails,
dotnet test --list-testswith the same filter writestest-results/main/listed-tests.txt, and the existing test-results artifact uploads it. The rows in flight are among those listed but not recorded in the TRX. Rows queued behind a hungNotInParallelkey are in that set too, so the listing narrows the search but does not name one row.Two command-line details, checked locally:
--list-testsgoes before--. Passed through to the test host, it fails the handshake ("reported execution mode 'discover', but 'dotnet test' expected 'run'").OpenCode.Sdk.Tools.Testslists 2033 tests, the same count as CI's main pass.If #447 recurs and the listing does not name the row, the next step is Microsoft.Testing.Extensions.HangDump, which names the tests in flight. It is a test-side package, so it needs a dependency decision; it is noted on #447.
Sizing
The timeout is about four times the slowest measured module. These are the slowest per-module times in two green master runs, 39022831816 and 39025318239:
OpenCode.Sdk.Testsnet472OpenCode.Sdk.Testsnet10.0OpenCode.Sdk.Tests5m51s × 4 is about 23 minutes, so 25 minutes. The election and native AOT passes take the same bound. Each runs a single module far shorter than the main pass.
Canon
docs/engineering/quality-gates.md, next to the roster paragraph, now states the bound, its measurement, exit code 7, and that the timeout does not name the rows in flight. The workflow comment points to it.Proof
f2e4457f(mastercf62b17a): all 8 steps, and 10,482 tests with 0 failed. The local gate does not runci.yml, so this shows only that nothing else regressed.Need help on this PR? Tag
@codesmith-botwith what you need.