Repository navigation
fix(backend): persist shutdown interruption for active scenario runs - #3817
Abhishek Jha (SpinachDigital) wants to merge 2 commits into
Conversation
|
Abhishek Jha (@SpinachDigital) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
1 similar comment
|
Abhishek Jha (@SpinachDigital) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
There was a problem hiding this comment.
🟡 Changes recommended
The shutdown fallback’s separate check and unconditional update can race with a concurrent resume, incorrectly overwriting the run’s IN_PROGRESS or COMPLETED state with FAILED.
1 open issue
What changed in this PR
Addresses #3746 by preserving shutdown-interruption status for active scenario runs, keeping them eligible for saved-config resume.
Changes:
- Adds a fallback to persist
FAILED/ScenarioInterruptedErrorwhen scenario cancellation has already writtenCANCELLED. - Adds a regression test that reproduces this write ordering with state-aware mocks.
| File | Description |
|---|---|
tests/unit/backend/test_scenario_run_service.py |
Tests shutdown persistence after the scenario writes CANCELLED. |
pyrit/backend/services/scenario_run_service.py |
Adds fallback persistence for shutdown cancellation intent. |
Review effort: Balanced
| if header is not None and header.scenario_run_state == ScenarioRunState.CANCELLED: | ||
| await self._memory.update_scenario_run_state_async( |

Fixes the shutdown path for active scenario runs (Closes #3746).
Root cause: shutdown_async sets the active run's cancellation intent to FAILED/ScenarioInterruptedError and then cancels its task, but Scenario.run_async persists CANCELLED when the cancellation propagates through it. The service handler's gated compare-and-swap write (CREATED/IN_PROGRESS only) then no-ops on the already-terminal row, so the run stays CANCELLED and loses saved-config resume eligibility. Queued runs were already correct, which matches the issue.
Fix: keep the gated write first (user-cancel path unchanged); when it no-ops and the stored row is CANCELLED but the service's cancellation intent differs, overwrite with the service's state. Other terminal states are left alone.
Tests (run locally before submit): new regression test test_shutdown_overwrites_scenario_cancelled_write_with_interrupted (scenario mock mirrors real Scenario.run_async: writes CANCELLED, then re-raises; try_update mock is CAS-faithful) plus the full test_scenario_run_service.py and test_scenario_resume.py files — 289 passed. Negative control: with the service fix stashed, the new test fails with the row staying CANCELLED. Note: ruff was not run here (binary absent from the local venv).
Closes #3746