[ParidelPooya]

Fixes #816.

Summary

Failed Lambda MicroVMs calls of microvm and microvm_session now keep their request ID in the error message, as <Operation> request ID: <id>.. A CreateMicrovmAuthToken ResourceNotFoundException now fails the job with MicrovmNotRunningError. A custom retry strategy still decides on the botocore error, with one exception. A ResourceNotFoundException from GetMicrovm, ResumeMicrovm, or CreateMicrovmAuthToken reaches the strategy as MicrovmStateError, with the botocore error as its __cause__. On main, GetMicrovm and ResumeMicrovm already did this. For CreateMicrovmAuthToken it is new: on main, the strategy received the ClientError. This ports aws/aws-durable-execution-sdk-js#1048, #1055, and #1060.

Why

1. The core SDK records a step failure as its class name and str(error). It does not record __cause__ or the botocore response.
2. The handler gets a StepError rebuilt from that record, on the first run and on replay.
3. The request ID is in response["ResponseMetadata"]["RequestId"].
4. So a support case had no request ID to quote.
5. A handler that catches MicrovmNotRunningError to start a new session missed a CreateMicrovmAuthToken not-found.

Changes

• aws_errors.py (new):
  • add_request_id returns a copy of the ClientError: the same class, the same response and operation_name, and the request ID in args. Its __cause__ is the original. The original keeps its message.
  • Only an error whose operation_name is the called operation gets the label. So an STS AssumeRole error from a credential provider keeps its message.
  • decide_on_original_error gives the step retry strategy the error that botocore raised. The core SDK builds the record before it calls the strategy, so the record keeps the label.
  • The map from each copy to its original holds weak references only: a weak key, and a weakref.ref to the original as the value. The reasons, in order:
    1. Under a deadline, a CreateMicrovmAuthToken call runs in the daemon thread of _run_with_limit. The thread function stores the copy in its closure.
    2. The traceback of the original holds the frame of that function. So the original references the copy.
    3. A strong value would keep its own key alive. So each labelled CreateMicrovmAuthToken failure would stay in memory for the life of the process, with its frames and the job body.
    4. The copy holds the original as its __cause__. So the weak value lives as long as the copy. When the value is dead, the strategy receives the copy.
    5. A WeakSet of copies, with __cause__ read as the original, would also work. Code can replace __cause__, and the strategy would then receive the wrong error. The weak value identifies the original by identity. So the map uses the weak value.
• lifecycle.py:
  • microvm_not_found has the neutral text: "A MicroVM lives at most its timeout plus 5 minutes, and never longer than 8 hours."
  • TERMINATING and TERMINATED name the GetMicrovm request ID.
  • Other GetMicrovm and ResumeMicrovm errors get the label.
  • MicrovmStateError stays in this module, because its fully qualified name is the recorded error type.
• request.py:
  • A CreateMicrovmAuthToken not-found raises MicrovmStateError. So a custom strategy receives MicrovmStateError for it, and not the ClientError.
  • Other errors, and a missing authToken, get the request ID.
• operation.py:
  • RunMicrovm and TerminateMicrovm errors, and a missing microvmId or endpoint, get the request ID.
  • step_retry_strategy wraps the strategy for both operations.
• Docs: the README error table, a request-ID paragraph, the ended-MicroVM section, and the docstrings. The retry_strategy docstring and the README name the not-found exception.

Why a copy, and not the JavaScript approach

1. The core SDK records the class name as ErrorType. So a wrapper of another class would change the history and fail isinstance checks.
2. Python can build an object of the same class. So the JavaScript reason to change the message in place does not apply.
3. The original error never changes. So a reused error object needs no map of original messages, and no message has to be restored after the retry strategy runs.

Testing

• request_ids_test.py: 16 scenarios, with and without an ID, on the local runner. They check the handler message on the first run and on replay, the history records, and the call counts. They also cover a real Stubber client, a real STS credential provider, throttled retries (5 calls, the last ID), reused error objects, a custom strategy that receives the botocore object, and 8 create_retry_strategy decisions.
• aws_errors_test.py: the helpers.
• Leak regression tests, 4 new:
  • request_ids_test.py: a session job on the local runner, with 5 throttled CreateMicrovmAuthToken calls. The test checks that each call ran in the daemon thread of _run_with_limit. After the runner closes, it calls gc.collect() and asserts that the 5 copies and the 5 originals are dead, and that the map is empty. The core SDK logs each failed step attempt with the error, and pytest keeps the log records until the test ends. So the test turns off the step logger.
  • aws_errors_test.py: the same check for one send_job call with a remaining time, for a copy that a caller frame keeps, as a handler closure can, and for a copy whose original is dead.
  • With the strong value of fd6cac2e restored, all 4 tests fail. With the weak value, all 4 pass.
• A leak probe ran 4 session jobs, each with a 1 MB job input and 5 throttled CreateMicrovmAuthToken calls, with -p no:logging, after a warm-up run. With the strong value: 20 map entries, 40 live ThrottlingException objects, and 21.7 MB still allocated by tracemalloc. With the weak value: 0 entries, 0 objects, and 0.0 MB.
• On main's source, 37 of the 53 request-ID tests of fd6cac2e fail. The STS tests and the no-ID tests of the other paths pass, so those messages are unchanged. Every strategy row has the same call count on main and on this branch. The request-ID file now imports aws_errors, which main does not have, so it no longer runs on main's source.
• 26 mutants of the new code at fd6cac2e all fail at least one test.
• Extras: 368 tests pass, with 100 % line and branch coverage. On main, 279 pass, with 100 %. The typecheck, format, and commit lint checks pass. The full repo suite passes: 5817 passed, 4 skipped.
• Not run against the live service. The answer of CreateMicrovmAuthToken for a removed MicroVM comes only from the API model.

[yaythomas]

[github-actions]

Codex AI review · Finding arf_v1_ih3txg2py4t7rvjdjuerpafiei

[P2] Check the operation before classifying a not-found error

On a later session job using SSO credentials, credential refresh can raise ResourceNotFoundException from GetRoleCredentials before CreateMicrovmAuthToken reaches Lambda. Checking only the error code incorrectly reports the MicroVM as removed, discards the credential failure's message from the checkpoint, and passes MicrovmStateError to custom retry strategies instead of the original ClientError. Require is_microvms_service_error("CreateMicrovmAuthToken", error) alongside the code check, and add a regression test where token creation propagates a credential-provider not-found error.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review

One P2 finding at packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/operation.py:564: credential-provider failures can be misclassified as removed MicroVMs. Tests were inspected but not executed under the review constraints.

Reviewed commit 037c58c307d43ddd549d2b592c15f02f0dfca913. Workflow run