bug: Prolonged test suite exceeding CLI default timeout of 5 minutes displaying false positives #979

Activity

1. Dnhem

   added

   bugSomething isn't workingSomething isn't working

   on Nov 12, 2024
2. sorccu commented on Nov 12, 2024

   

   sorccu

   on Nov 12, 2024

   Member

   More actions

   Thanks for the report, I'll look into this as soon as possible.
3. everzel commented on Nov 26, 2024

   

   everzel

   on Nov 26, 2024

   More actions

   @sorccu Hi, any updates?
4. sorccu

   added 2 commits that reference this issue on Oct 26, 2026

   fix(reporters): show CLI-side timeouts as timed out in all reporters

   ...

   61800c4

   fix(reporters): report CLI-side timeouts as timed out, not failed [RE…

   ...

   bedb773
5. sorccu

   closed this as completedin #1582on Oct 26, 2026
6. sorccu commented on Oct 26, 2026

   

   sorccu

   on Oct 26, 2026

   Member

   More actions

   Thanks for the report, and sorry it sat for so long.

   This has been addressed in a few steps since CLI version 4.9.1:

   • The timeout is longer and per check. --timeout now applies to each check separately rather than to the whole run. It defaults to 600 seconds (1200 for Playwright checks), restarts when a check starts running and on each retry, and no longer counts time a check spends queued before it starts. A long test suite whose individual checks finish in time no longer hits the timeout.
   • Reports explain the mismatch. When the CLI does stop waiting for a check, the output lists it under "Execution Errors" and says the test session may show a different status for it.
   • Timed-out checks are no longer shown as failures. With fix(reporters): report CLI-side timeouts as timed out, not failed [RED-1276] [ship] #1582, a check the CLI stopped waiting for is reported as timed out: ⏱️ Timed out in the GitHub markdown, ⏱ and an "N timed out" count in the list and ci output, T in the dot reporter, and "result": "Timed out" in the json report. The run still exits with code 1, because the CLI never received a result for that check.

   The first two are in current releases. The last one will close this issue when #1582 merges and will ship in the next release. If you still need more time for slow checks, raise --timeout.