Skip to content

fix(microvm-worker): state that jobs are accepted after an unknown outcome whose refusal ended - #1069

Merged
ParidelPooya merged 3 commits into
mainfrom
fix/microvm-worker-unknown-outcome-warning
Oct 23, 2026
Merged

ParidelPooya merged 3 commits into
mainfrom
fix/microvm-worker-unknown-outcome-warning

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1056.

Summary

The MicroVM worker logs a warning when its own SuspendMicrovm call has an unknown outcome. Without a terminate hook, the warning said that the worker refuses jobs. That was false when the refusal had already ended during the call. This PR changes only which text the worker logs. The behavior is unchanged.

Why

  1. The warning said: "The worker refuses jobs until the MicroVM resumes, or for 30 seconds."
  2. A resume hook during the call ends the refusal of jobs.
  3. The 30-second limit of a suspend hook's refusal can also end during the call. This needs a client from createMicrovmsClient that runs past the worker's 10-second abort.
  4. After an unknown outcome, the worker starts a new refusal only while suspending is still set.
  5. So in these sequences the worker accepts jobs. A job right after the warning gets HTTP 202.
  6. The likely sequence is the first one: the worker's own call suspended the MicroVM, and its answer was lost across the freeze.

Changes

  • failedCallWarning reads suspending for an unknown outcome. The catch block calls it before the unknown branch acts on the state.
  • When the refusal already ended, the warning says: "the suspend call for the idle MicroVM had an unknown outcome. The refusal of jobs already ended during the call, on a resume hook or after 30 seconds. So the worker accepts jobs. It tries the call again one idle time after the call and every job end."
  • The idle time cannot start while a job runs. A job can start during the call after a resume hook. So the text names the call and every job end, as the refused-for-now text for the same state does.
  • A closed worker starts no idle time. So after close() during the call, the same case says: "… So the worker accepts jobs. The worker is closed, so it does not try the call again."
  • The terminating text and the normal unknown text are unchanged.
  • The JSDoc of failedCallWarning lists the new cases.
  • README item 8 and the listener JSDoc item 8 end with: "A resume hook during the call, or the end of a suspend hook's 30 seconds during the call, has already ended the refusal. The worker then accepts jobs at once, and tries the call again one idle time after the call and every job end."
  • The Python worker has the same text. fix(microvm-worker): state that the refusal ended aws-durable-execution-sdk-python#817 makes the same change, with identical texts and conditions.

Testing

  • 12 new tests in auto-suspend.test.ts. The three sequences from the issue each assert the full text, one warning, HTTP 202 for a job right after, and the next call one idle time after that job ends.
  • Other tests cover the idle time that starts at the call end, the closed variant, a resume hook and then a terminate hook, and a guard that keeps the old text with no hook, with a suspend hook still in effect, and after close() alone.
  • A job that still runs at the outcome: the warning has the new text, and the next call comes one idle time after the job ends.
  • A resume hook and then a suspend hook during the call, and two suspend hooks with the call ending during the second refusal: both keep the refusal text and refuse jobs for 30 seconds after the call.
  • 6 of the new tests fail on main. With the old texts expected, all tests pass on main. So calls, statuses, and timings are unchanged.
  • Removing or inverting each new condition fails at least one test.
  • npm test in the package passes 292 tests. Coverage rose from 96.57 % to 96.59 % of statements, and from 92.53 % to 92.61 % of branches.
  • npm run build, the package lint, the root npm run lint, and npm run format:check pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…tcome whose refusal ended

The worker logs a warning when its own SuspendMicrovm call has an
unknown outcome. Without a terminate hook, the warning said: "The
worker refuses jobs until the MicroVM resumes, or for 30 seconds."

1. A resume hook during the call ends the refusal of jobs.
2. The 30-second limit of a suspend hook's refusal can also end during
   the call.
3. After an unknown outcome, the worker starts a new refusal only while
   `suspending` is still set.
4. So after these sequences, the worker accepts jobs. A job right after
   the warning gets HTTP 202.
5. So the warning stated a refusal that had already ended.

`failedCallWarning` now reads `suspending` for an unknown outcome. The
catch block calls it before the unknown branch acts on the state.

- When the refusal already ended, the warning says so, and says that
  the worker accepts jobs and that the idle time starts again.
- After close() during the call, the same case says that the worker is
  closed and does not try the call again. A closed worker starts no
  idle time.
- The terminating text and the normal unknown text are unchanged.
- Behavior is unchanged.

Fixes #1056
ParidelPooya added a commit to aws/aws-durable-execution-sdk-python that referenced this pull request Oct 23, 2026
The unknown-outcome warning for a refusal that ended during the call
said: "So the worker accepts jobs, and the idle time starts again."

1. A resume hook during the call ends the refusal of jobs.
2. A job can then start during the call.
3. A running job cancels the idle time.
4. So the idle time does not start at the outcome. It starts when the
   call and every job have ended.
5. So the text now says: "So the worker accepts jobs. It tries the call
   again one idle time after the call and every job end." The text for
   a refusal for now after a suspend hook ends the same way.

A new test covers this sequence. A job runs at the outcome. The next
call comes one idle time after the last job ends.

README item 8 and the listener docstring item 8 now state this case.
The `_failed_call_warning` docstring item 8 says the same.

The comment in the decision block said that the text matches the
decision whatever another thread does. That holds for a rejection and
a refusal for now, because the same block decides. For an unknown
outcome, the block at the end of `_call_suspend` decides whether the
30-second refusal starts. The comment now says so. Issue #820 tracks
the window between the two blocks.

Behavior is unchanged. The texts match the JavaScript worker's change
in aws/aws-durable-execution-sdk-js#1069.
…wn outcome whose refusal ended

The ended text of the unknown-outcome warning said: "So the worker
accepts jobs, and the idle time starts again."

1. A resume hook during the call ends the refusal of jobs.
2. So a job can start during the call.
3. startIdleTime starts no idle time while a job runs.
4. So the idle time starts when the call and every job have ended, not
   when the warning is logged.
5. So the text now says: "So the worker accepts jobs. It tries the call
   again one idle time after the call and every job end." The
   refused-for-now text for the same state uses the same sentence.

README item 8 and the listener JSDoc item 8 now state this case. A resume
hook or the end of a suspend hook's 30 seconds during the call has already
ended the refusal. The worker then accepts jobs at once.

The second line of item 10 in the failedCallWarning JSDoc is now aligned
under its text.

Three new tests cover sequences that had no test:

- A job still runs at the outcome. The next call comes one idle time
  after the job ends.
- A resume hook and then a suspend hook arrive during the call. The
  warning keeps the refusal text.
- Two suspend hooks arrive during the call, and the call ends during the
  second hook's refusal. The warning keeps the refusal text.

Behavior is unchanged. aws/aws-durable-execution-sdk-python#817 makes the
same change.

Refs #1056

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

ParidelPooya added a commit to aws/aws-durable-execution-sdk-python that referenced this pull request Oct 23, 2026
* fix(microvm-worker): state that the refusal ended

The worker logs a warning when its own SuspendMicrovm call has an
unknown outcome. Without a terminate hook, the warning said: "The
worker refuses jobs until the MicroVM resumes, or for 30 seconds."

1. A resume hook during the call ends the refusal of jobs.
2. The 30-second limit of a suspend hook's refusal can also end during
   the call.
3. After an unknown outcome, the worker starts a new refusal only while
   `_suspending` is still set.
4. So after these sequences, the worker accepts jobs. A job right after
   the warning gets HTTP 202.
5. So the warning stated a refusal that had already ended.

`_failed_call_warning` now reads `suspending` for an unknown outcome.
The caller reads it under the lock, as before.

- When the refusal already ended, the warning says so. It also says
  that the worker accepts jobs and that the idle time starts again.
- After close() during the call, the same case says that the worker is
  closed and does not try the call again. A closed worker starts no
  idle time.
- The terminating text and the normal unknown text are unchanged.
- Behavior is unchanged.

The texts and conditions match the JavaScript worker's fix for
aws/aws-durable-execution-sdk-js#1056.

Fixes #806

* fix(microvm-worker): state when the call repeats

The unknown-outcome warning for a refusal that ended during the call
said: "So the worker accepts jobs, and the idle time starts again."

1. A resume hook during the call ends the refusal of jobs.
2. A job can then start during the call.
3. A running job cancels the idle time.
4. So the idle time does not start at the outcome. It starts when the
   call and every job have ended.
5. So the text now says: "So the worker accepts jobs. It tries the call
   again one idle time after the call and every job end." The text for
   a refusal for now after a suspend hook ends the same way.

A new test covers this sequence. A job runs at the outcome. The next
call comes one idle time after the last job ends.

README item 8 and the listener docstring item 8 now state this case.
The `_failed_call_warning` docstring item 8 says the same.

The comment in the decision block said that the text matches the
decision whatever another thread does. That holds for a rejection and
a refusal for now, because the same block decides. For an unknown
outcome, the block at the end of `_call_suspend` decides whether the
30-second refusal starts. The comment now says so. Issue #820 tracks
the window between the two blocks.

Behavior is unchanged. The texts match the JavaScript worker's change
in aws/aws-durable-execution-sdk-js#1069.
@ParidelPooya
ParidelPooya marked this pull request as ready for review October 23, 2026 21:39
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 21:40 — with GitHub Actions Inactive
@ParidelPooya
ParidelPooya temporarily deployed to ai-pr-review-runtime October 23, 2026 21:40 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

Codex AI review

No actionable findings. The SDK invariants and protected contract tests are unchanged. Tests were inspected but not run because repository code execution was prohibited; runtime behavior remains unverified.

Reviewed commit 4d5aa93bea81a8355f0e468b1a2b212ca53dcc34. Workflow run

@ParidelPooya
ParidelPooya merged commit 962f2bd into main Oct 23, 2026
34 of 36 checks passed
@ParidelPooya
ParidelPooya deleted the fix/microvm-worker-unknown-outcome-warning branch October 23, 2026 22:00
ParidelPooya added a commit that referenced this pull request Oct 24, 2026
## Summary

After a failed SuspendMicrovm call, the worker logs one warning per
outcome, with the state as fields. The 10 texts of `failedCallWarning`
are removed. The behavior is unchanged.

## Why

1. `failedCallWarning` picked one of 10 texts from the outcome and four
state flags.
2. The texts change no behavior.
3. #1049 and #1069 here, and Python #799 and #817, only fixed texts that
stated something false.
4. Each text had to stay word for word identical in both SDKs.
5. So the message now names only the outcome, and fields carry the
state. A field cannot state a false sentence.

This is item D2 of a review of the MicroVM safeguards. The Python worker
makes the same change in aws/aws-durable-execution-sdk-python#831.

## Changes

- Messages:
- rejected: "could not suspend the idle MicroVM, because the service
rejected the call. A missing permission needs lambda:SuspendMicrovm on
the MicroVM image in its execution role."
- refused for now: "could not suspend the idle MicroVM, because the
service refused the call for now."
- unknown: "the suspend call for the idle MicroVM had an unknown
outcome."
- Fields, in this order: `microvmId`, `error`, `outcome`,
`suspendHookArrived`, `terminating`, `closed`, `refusingJobs`.
- The worker logs the warning after it has acted on the outcome. So
`refusingJobs` is the state that follows from the outcome. It is
`terminating || suspending`, the two states in which the job route
answers 503.
- The 30-second wait after a call refused for now is unchanged.
- README and listener TSDoc item 9 describe the warning once, with all
seven fields. The CHANGELOG has a `Changed` entry, because a log query
that matched an old text must now match the new message or `outcome`.
- One log order changes. After a rejection or a refusal for now,
deferred `run` hook jobs start before the warning. So their "job
started" line now comes before the warning. Python keeps the warning
first. No test or documented behavior depends on this order.
- For an unknown outcome, the grace timer now starts before the worker
destroys its client. So a `destroy()` that throws no longer leaves the
refusal in place until the `resume` hook.

## Testing

- Every existing scenario is kept. Each asserts the message, the fields,
and one warning per failed call. The 202/503 and timing assertions are
unchanged.
- Reading `refusingJobs` before the outcome is handled fails 28 tests.
- Two tests were added for the default JSON logger and an unknown hook
name, to keep the coverage.
- Worker `npm test`: 294 tests pass. Coverage rose from 96.59 % to 96.78
% of statements. `npm run lint` and `npm run format:check` pass.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

This branch was previously deployed

1 inactive deployment
ai-pr-review-runtime — 4d5aa93b Deployed Oct 23, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #1767
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MicroVM worker: the unknown-outcome warning says jobs are refused after the refusal ended

2 participants