Repository navigation
fix(microvm-worker): state that jobs are accepted after an unknown outcome whose refusal ended - #1069
Merged
Merged
Conversation
…tcome whose refusal ended The worker logs a warning when its own SuspendMicrovm call has an unknown outcome. Without a terminate hook, the warning said: "The worker refuses jobs until the MicroVM resumes, or for 30 seconds." 1. A resume hook during the call ends the refusal of jobs. 2. The 30-second limit of a suspend hook's refusal can also end during the call. 3. After an unknown outcome, the worker starts a new refusal only while `suspending` is still set. 4. So after these sequences, the worker accepts jobs. A job right after the warning gets HTTP 202. 5. So the warning stated a refusal that had already ended. `failedCallWarning` now reads `suspending` for an unknown outcome. The catch block calls it before the unknown branch acts on the state. - When the refusal already ended, the warning says so, and says that the worker accepts jobs and that the idle time starts again. - After close() during the call, the same case says that the worker is closed and does not try the call again. A closed worker starts no idle time. - The terminating text and the normal unknown text are unchanged. - Behavior is unchanged. Fixes #1056
ParidelPooya
added a commit
to aws/aws-durable-execution-sdk-python
that referenced
this pull request
Oct 23, 2026
The unknown-outcome warning for a refusal that ended during the call said: "So the worker accepts jobs, and the idle time starts again." 1. A resume hook during the call ends the refusal of jobs. 2. A job can then start during the call. 3. A running job cancels the idle time. 4. So the idle time does not start at the outcome. It starts when the call and every job have ended. 5. So the text now says: "So the worker accepts jobs. It tries the call again one idle time after the call and every job end." The text for a refusal for now after a suspend hook ends the same way. A new test covers this sequence. A job runs at the outcome. The next call comes one idle time after the last job ends. README item 8 and the listener docstring item 8 now state this case. The `_failed_call_warning` docstring item 8 says the same. The comment in the decision block said that the text matches the decision whatever another thread does. That holds for a rejection and a refusal for now, because the same block decides. For an unknown outcome, the block at the end of `_call_suspend` decides whether the 30-second refusal starts. The comment now says so. Issue #820 tracks the window between the two blocks. Behavior is unchanged. The texts match the JavaScript worker's change in aws/aws-durable-execution-sdk-js#1069.
…wn outcome whose refusal ended The ended text of the unknown-outcome warning said: "So the worker accepts jobs, and the idle time starts again." 1. A resume hook during the call ends the refusal of jobs. 2. So a job can start during the call. 3. startIdleTime starts no idle time while a job runs. 4. So the idle time starts when the call and every job have ended, not when the warning is logged. 5. So the text now says: "So the worker accepts jobs. It tries the call again one idle time after the call and every job end." The refused-for-now text for the same state uses the same sentence. README item 8 and the listener JSDoc item 8 now state this case. A resume hook or the end of a suspend hook's 30 seconds during the call has already ended the refusal. The worker then accepts jobs at once. The second line of item 10 in the failedCallWarning JSDoc is now aligned under its text. Three new tests cover sequences that had no test: - A job still runs at the outcome. The next call comes one idle time after the job ends. - A resume hook and then a suspend hook arrive during the call. The warning keeps the refusal text. - Two suspend hooks arrive during the call, and the call ends during the second hook's refusal. The warning keeps the refusal text. Behavior is unchanged. aws/aws-durable-execution-sdk-python#817 makes the same change. Refs #1056
ParidelPooya
added a commit
to aws/aws-durable-execution-sdk-python
that referenced
this pull request
Oct 23, 2026
* fix(microvm-worker): state that the refusal ended The worker logs a warning when its own SuspendMicrovm call has an unknown outcome. Without a terminate hook, the warning said: "The worker refuses jobs until the MicroVM resumes, or for 30 seconds." 1. A resume hook during the call ends the refusal of jobs. 2. The 30-second limit of a suspend hook's refusal can also end during the call. 3. After an unknown outcome, the worker starts a new refusal only while `_suspending` is still set. 4. So after these sequences, the worker accepts jobs. A job right after the warning gets HTTP 202. 5. So the warning stated a refusal that had already ended. `_failed_call_warning` now reads `suspending` for an unknown outcome. The caller reads it under the lock, as before. - When the refusal already ended, the warning says so. It also says that the worker accepts jobs and that the idle time starts again. - After close() during the call, the same case says that the worker is closed and does not try the call again. A closed worker starts no idle time. - The terminating text and the normal unknown text are unchanged. - Behavior is unchanged. The texts and conditions match the JavaScript worker's fix for aws/aws-durable-execution-sdk-js#1056. Fixes #806 * fix(microvm-worker): state when the call repeats The unknown-outcome warning for a refusal that ended during the call said: "So the worker accepts jobs, and the idle time starts again." 1. A resume hook during the call ends the refusal of jobs. 2. A job can then start during the call. 3. A running job cancels the idle time. 4. So the idle time does not start at the outcome. It starts when the call and every job have ended. 5. So the text now says: "So the worker accepts jobs. It tries the call again one idle time after the call and every job end." The text for a refusal for now after a suspend hook ends the same way. A new test covers this sequence. A job runs at the outcome. The next call comes one idle time after the last job ends. README item 8 and the listener docstring item 8 now state this case. The `_failed_call_warning` docstring item 8 says the same. The comment in the decision block said that the text matches the decision whatever another thread does. That holds for a rejection and a refusal for now, because the same block decides. For an unknown outcome, the block at the end of `_call_suspend` decides whether the 30-second refusal starts. The comment now says so. Issue #820 tracks the window between the two blocks. Behavior is unchanged. The texts match the JavaScript worker's change in aws/aws-durable-execution-sdk-js#1069.
ParidelPooya
marked this pull request as ready for review
October 23, 2026 21:39
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 21:40 — with
GitHub Actions
Inactive
ParidelPooya
had a problem deploying
to
ai-pr-review-runtime
October 23, 2026 21:40 — with
GitHub Actions
Error
ParidelPooya
temporarily deployed
to
ai-pr-review-runtime
October 23, 2026 21:40 — with
GitHub Actions
Inactive
Codex AI reviewNo actionable findings. The SDK invariants and protected contract tests are unchanged. Tests were inspected but not run because repository code execution was prohibited; runtime behavior remains unverified. Reviewed commit |
ParidelPooya
added a commit
that referenced
this pull request
Oct 24, 2026
## Summary After a failed SuspendMicrovm call, the worker logs one warning per outcome, with the state as fields. The 10 texts of `failedCallWarning` are removed. The behavior is unchanged. ## Why 1. `failedCallWarning` picked one of 10 texts from the outcome and four state flags. 2. The texts change no behavior. 3. #1049 and #1069 here, and Python #799 and #817, only fixed texts that stated something false. 4. Each text had to stay word for word identical in both SDKs. 5. So the message now names only the outcome, and fields carry the state. A field cannot state a false sentence. This is item D2 of a review of the MicroVM safeguards. The Python worker makes the same change in aws/aws-durable-execution-sdk-python#831. ## Changes - Messages: - rejected: "could not suspend the idle MicroVM, because the service rejected the call. A missing permission needs lambda:SuspendMicrovm on the MicroVM image in its execution role." - refused for now: "could not suspend the idle MicroVM, because the service refused the call for now." - unknown: "the suspend call for the idle MicroVM had an unknown outcome." - Fields, in this order: `microvmId`, `error`, `outcome`, `suspendHookArrived`, `terminating`, `closed`, `refusingJobs`. - The worker logs the warning after it has acted on the outcome. So `refusingJobs` is the state that follows from the outcome. It is `terminating || suspending`, the two states in which the job route answers 503. - The 30-second wait after a call refused for now is unchanged. - README and listener TSDoc item 9 describe the warning once, with all seven fields. The CHANGELOG has a `Changed` entry, because a log query that matched an old text must now match the new message or `outcome`. - One log order changes. After a rejection or a refusal for now, deferred `run` hook jobs start before the warning. So their "job started" line now comes before the warning. Python keeps the warning first. No test or documented behavior depends on this order. - For an unknown outcome, the grace timer now starts before the worker destroys its client. So a `destroy()` that throws no longer leaves the refusal in place until the `resume` hook. ## Testing - Every existing scenario is kept. Each asserts the message, the fields, and one warning per failed call. The 202/503 and timing assertions are unchanged. - Reading `refusingJobs` before the outcome is handled fails 28 tests. - Two tests were added for the default JSON logger and an unknown hook name, to keep the coverage. - Worker `npm test`: 294 tests pass. Coverage rose from 96.59 % to 96.78 % of statements. `npm run lint` and `npm run format:check` pass. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1056.
Summary
The MicroVM worker logs a warning when its own SuspendMicrovm call has an unknown outcome. Without a
terminatehook, the warning said that the worker refuses jobs. That was false when the refusal had already ended during the call. This PR changes only which text the worker logs. The behavior is unchanged.Why
resumehook during the call ends the refusal of jobs.suspendhook's refusal can also end during the call. This needs a client fromcreateMicrovmsClientthat runs past the worker's 10-second abort.suspendingis still set.Changes
failedCallWarningreadssuspendingfor an unknown outcome. The catch block calls it before the unknown branch acts on the state.resumehook. So the text names the call and every job end, as the refused-for-now text for the same state does.close()during the call, the same case says: "… So the worker accepts jobs. The worker is closed, so it does not try the call again."failedCallWarninglists the new cases.resumehook during the call, or the end of asuspendhook's 30 seconds during the call, has already ended the refusal. The worker then accepts jobs at once, and tries the call again one idle time after the call and every job end."Testing
auto-suspend.test.ts. The three sequences from the issue each assert the full text, one warning, HTTP 202 for a job right after, and the next call one idle time after that job ends.resumehook and then aterminatehook, and a guard that keeps the old text with no hook, with asuspendhook still in effect, and afterclose()alone.resumehook and then asuspendhook during the call, and twosuspendhooks with the call ending during the second refusal: both keep the refusal text and refuse jobs for 30 seconds after the call.main. With the old texts expected, all tests pass onmain. So calls, statuses, and timings are unchanged.npm testin the package passes 292 tests. Coverage rose from 96.57 % to 96.59 % of statements, and from 92.53 % to 92.61 % of branches.npm run build, the package lint, the rootnpm run lint, andnpm run format:checkpass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.