fix(extras): keep AWS request IDs in MicroVM errors

packages/aws-durable-execution-sdk-python-extras/README.md

@@ -95,7 +95,7 @@ Every MicroVM failure is a `MicrovmError`. Catch the base class for all of them,
 |---|---|
 | `MicrovmLaunchError` | `RunMicrovm` failed after all retries. When the last attempt timed out or lost its connection, the service may have created the MicroVM anyway, and it then runs the job. So running the job again after this error can run it twice. |
 | `MicrovmDeliveryError` | HTTP delivery failed after all retries, or the route answered a 4xx status such as 404, or a redirect. |
-| `MicrovmNotRunningError` | A session job found its MicroVM terminating, terminated, or removed. It is a `MicrovmDeliveryError`. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). |
+| `MicrovmNotRunningError` | A `MicrovmDeliveryError`: the MicroVM is terminating, terminated, or removed. It is not retried. A session that checks the MicroVM state before each job reports it. Any job over HTTP also reports it when `CreateMicrovmAuthToken` reports the MicroVM removed. See [Suspending an idle MicroVM](#suspending-an-idle-microvm). |
 | `MicrovmJobFailedError` | The job handler in the MicroVM failed, or the MicroVM sent a result that is not valid JSON. The message includes the error type that the MicroVM reported. |
 | `MicrovmTimeoutError` | No result arrived within `timeout`, or no heartbeat within `heartbeat_timeout`. A MicroVM that crashed ends this way. |
 
@@ -115,6 +115,8 @@ except MicrovmError:
 
 The class is the same on the first run and on every replay, so code that branches on it stays deterministic. A `microvm` call inside your own child context, or inside a `map` or `parallel` item, reaches the code around that context as the SDK's `ChildContextError`. Its `__cause__` is the MicroVM error, so check `isinstance(error.__cause__, MicrovmTimeoutError)` there. An invalid name or config raises the SDK's `ValidationError`, and an input that is not JSON-serializable raises `TypeError`, before any durable operation starts. `Duration` raises the same `ValidationError` for a negative value.
 
+A failed AWS call keeps its request ID in the error message, in the form `<Operation> request ID: <id>.`, for example `RunMicrovm request ID: <id>.`. A support case can quote it. This applies to `RunMicrovm`, `GetMicrovm`, `ResumeMicrovm`, `CreateMicrovmAuthToken`, and `TerminateMicrovm`. The SDK checkpoints the message of a failed step, and not its `__cause__`. So the request ID is in the message, the handler sees it on the first run and on replay, and the execution history keeps it. A failed `TerminateMicrovm` call does not fail the operation. So a `TerminateMicrovm` request ID appears only in the terminate log entry and in the terminate step record. A message has no request ID when the answer carries none, for example after a connection failure. A message also has no request ID when the error is not the answer of the Lambda MicroVMs call. For example, a `client` with role-assuming credentials calls STS `AssumeRole` inside each call, and an STS error carries the request ID of STS. The operation recognizes such an error by its `operation_name`. The step raises a copy of the botocore error, of the same class, with the same `response`, and with the request ID in its message. So `default_microvm_retry_strategy` decides as before. A custom `retry_strategy` receives the error that botocore raised, without the request ID. So a strategy that matches the message, such as `create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))`, decides on the message of the service. A `ResourceNotFoundException` from `GetMicrovm`, `ResumeMicrovm`, or `CreateMicrovmAuthToken` is the exception. It means that the MicroVM no longer exists. So the operation raises `MicrovmStateError` instead, and the strategy receives that error. Its `__cause__` is the botocore error. An error that the operation creates itself, such as `MicrovmStateError` or "RunMicrovm returned no microvmId", has the request ID in its message when the strategy receives it. A custom client can raise one error object for several calls. Each step message then names only its own call, and the object keeps its message.
+
 When the error escapes the handler, the execution's error type is the class's fully-qualified name, such as `aws_durable_execution_sdk_python_extras.microvm.errors.MicrovmJobFailedError`. The SDK uses that name to rebuild the class on replay. The JavaScript operation reports the short name, `MicrovmJobFailedError`. So a rule that matches the error type of both SDKs, such as a Step Functions `Catch` or a metric filter, must match both forms. For example, match on the suffix.
 
 ### Retries in two tiers
@@ -243,7 +245,11 @@ For a session that launched with either setting, each request step calls `GetMic
 
 The worker reads its idle time once, from the run hook payload of the launch. `RunMicrovm` sets the idle policy once, at the launch. So a deploy that changes `auto_suspend_on_idle` or `idle_policy` does not change what a running MicroVM does. The resume mode says whether each job checks the MicroVM state, and resumes a suspended MicroVM, before delivery. The launch checkpoint records the resume mode in its `resumeBeforeDelivery` field. Replay follows the recorded resume mode, not the current config. So a deploy changes the resume mode only for sessions that launch after it.
 
-A MicroVM that is terminating, terminated, or removed fails the job at once with `MicrovmNotRunningError`, a subclass of `MicrovmDeliveryError`. A retry finds the same state, so the job is not retried. A session that launched with neither setting makes no `GetMicrovm` call. A terminated MicroVM's endpoint answers 502, and the request step retries a 502 as a MicroVM that is still starting. So the job fails with a plain `MicrovmDeliveryError`, after the retries of both tiers.
+A MicroVM lives at most its `timeout` plus 5 minutes, and never longer than 8 hours. What the next `vm.invoke` reports after the MicroVM ends depends on the state check:
+
+- For a session that launched with `auto_suspend_on_idle=True`, or with an `idle_policy`, the request step calls `GetMicrovm` before each job. A MicroVM that is terminating, terminated, or removed then fails the job at once with `MicrovmNotRunningError`, a subclass of `MicrovmDeliveryError`. A retry finds the same state, so the job is not retried. The message ends with the request ID of the call that reported the state, for example `GetMicrovm request ID: <id>.`.
+- Every session, and every `microvm` job over HTTP, calls `CreateMicrovmAuthToken` before the first request, and again after a 401 or 403. The API model lists `ResourceNotFoundException` for that call. That answer fails the job at once with `MicrovmNotRunningError`, whose message ends with `CreateMicrovmAuthToken request ID: <id>.`. The live service's answer for a removed MicroVM is not verified.
+- A session that launched with neither setting makes no `GetMicrovm` call. When `CreateMicrovmAuthToken` returns a token, the request goes to the endpoint. A terminated MicroVM's endpoint answers 502, and the request step retries a 502 as a MicroVM that is still starting. So the job fails with a plain `MicrovmDeliveryError`, after the retries of both tiers.
 
 Suspending is off by default, because it needs more permissions and changes what the MicroVM does between jobs:
 

packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/aws_errors.py

@@ -0,0 +1,200 @@
+# SPDX-FileCopyrightText: 2026-present Amazon.com, Inc. or its affiliates.
+#
+# SPDX-License-Identifier: Apache-2.0
+"""The request IDs of the AWS calls that the MicroVM operations make.
+
+The core SDK checkpoints a step failure as its type and its message,
+``str(error)``. It does not checkpoint ``__cause__``, or the ``response`` of a
+botocore ``ClientError``. The handler then receives a ``StepError`` rebuilt
+from that record, on the first run and on replay. The request ID of a failed
+call is in ``error.response["ResponseMetadata"]["RequestId"]``. So only the
+message can carry the request ID to the handler and to the execution history.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Mapping
+from typing import Literal
+
+from aws_durable_execution_sdk_python.retries import RetryDecision
+from botocore.exceptions import ClientError
+
+
+MicrovmApiOperation = Literal[
+    "RunMicrovm",
+    "GetMicrovm",
+    "ResumeMicrovm",
+    "CreateMicrovmAuthToken",
+    "TerminateMicrovm",
+]
+"""The Lambda MicroVMs API operations that the MicroVM operations call."""
+
+
+def error_code(error: ClientError) -> str | None:
+    """Return the error code of a botocore ``ClientError``."""
+    return error.response.get("Error", {}).get("Code")
+
+
+def is_microvms_service_error(operation: MicrovmApiOperation, error: object) -> bool:
+    """Return whether ``error`` is the answer of the Lambda MicroVMs call
+    ``operation``.
+
+    1. boto3 resolves the credentials inside each call.
+    2. A credential provider can call another service there, such as STS
+       ``AssumeRole`` for an assumed role.
+    3. When that call fails, the MicroVMs call raises the ``ClientError`` of
+       the other service. That error carries the request ID of the other
+       service.
+    4. botocore sets ``operation_name`` to the name of the API operation that
+       answered. An STS error names ``AssumeRole``, not ``RunMicrovm``.
+    5. So only a ``ClientError`` whose ``operation_name`` is ``operation``
+       gets the name of a Lambda MicroVMs operation next to its request ID.
+    """
+    return isinstance(error, ClientError) and error.operation_name == operation
+
+
+def request_id_sentence(operation: MicrovmApiOperation, response: object) -> str:
+    """Return ``<Operation> request ID: <id>.`` for an AWS call.
+
+    ``response`` is the output of a boto3 call, or the ``response`` of a
+    ``ClientError``. Both carry the request ID in
+    ``ResponseMetadata.RequestId``. A custom client can return an output
+    without it. So the function returns ``""`` when ``response`` has no
+    request ID that is a non-empty string.
+    """
+    metadata = (
+        response.get("ResponseMetadata") if isinstance(response, Mapping) else None
+    )
+    request_id = metadata.get("RequestId") if isinstance(metadata, Mapping) else None
+    if isinstance(request_id, str) and request_id:
+        return f"{operation} request ID: {request_id}."
+    return ""
+
+
+def with_request_id(
+    message: str, operation: MicrovmApiOperation, response: object
+) -> str:
+    """Return ``message`` followed by the sentence of
+    :func:`request_id_sentence`.
+
+    The function returns ``message`` unchanged when ``response`` has no
+    request ID. A service message often ends without a period, as in
+    ``MicroVM not found: mvm-1``. So a message that does not end a sentence
+    gets a period before the request ID sentence.
+    """
+    sentence = request_id_sentence(operation, response)
+    if not sentence:
+        return message
+    if not message:
+        return sentence
+    end = "" if message.endswith((".", "!", "?")) else "."
+    return f"{message}{end} {sentence}"
+
+
+def add_request_id(operation: MicrovmApiOperation, error: ClientError) -> ClientError:
+    """Return a copy of ``error`` whose message ends with the request ID of
+    the failed call, or ``error`` itself.
+
+    The step raises the result. The caller raises it without ``from``: the
+    copy already has ``error`` as its ``__cause__``, and ``error`` must not
+    become its own cause.
+
+    The function makes a copy, and it does not change ``error``. The
+    reasons, in order:
+
+    1. The core SDK records ``str()`` of the raised error as the message,
+       and the fully-qualified name of its class as the error type.
+    2. So the raised error must have the request ID in ``str()``, and the
+       class of ``error``. A wrapper of another class would change the
+       recorded error type.
+    3. The copy has the class of ``error``, the same ``response``, and the
+       same ``operation_name``. Only ``args``, ``__cause__``, and
+       ``_original_error`` differ. ``args`` gives ``str()``. So
+       :func:`default_microvm_retry_strategy`, which reads the class, the
+       error code, and the HTTP status, decides the same way.
+    4. A fake or a custom client can raise one error object for several
+       calls. ``error`` keeps its message, and each call builds its copy from
+       that message. So the message names only the call that raised it.
+    5. A custom strategy can match the message, for example with
+       ``create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))``.
+       The longer message could change its decision. So the copy holds
+       ``error`` in its ``_original_error`` attribute, and
+       :func:`decide_on_original_error` gives the strategy ``error`` instead
+       of the copy. The core SDK builds the step record before it calls the
+       strategy. So the record keeps the message with the request ID.
+
+    The copy is made without ``__init__``. A subclass of ``ClientError`` can
+    take other arguments, so ``__init__`` could fail. The copy takes
+    ``__dict__`` of ``error``, which holds ``response``, ``operation_name``,
+    and any notes. ``_original_error`` is set in the same ``__dict__``. When
+    ``error`` is itself a copy, its ``_original_error`` is replaced with
+    ``error``. So the attribute always names the error that this call
+    received.
+
+    ``raise ... from ...`` sets ``__cause__``, and code can set
+    ``__cause__`` too. Neither changes ``_original_error``. So the strategy
+    receives ``error`` even after ``__cause__`` changes. No module state
+    holds an error. The copy holds ``error``. So ``error`` lives at least as
+    long as the copy, and the garbage collector frees both when nothing
+    else refers to them.
+
+    The function returns ``error`` itself in these cases:
+
+    - ``error`` is not the answer of ``operation``. See
+      :func:`is_microvms_service_error`.
+    - ``error`` has no request ID.
+    - The class of ``error`` builds ``str()`` from something other than
+      ``args``. The copy would not show the request ID.
+    """
+    if not is_microvms_service_error(operation, error):
+        return error
+    message = with_request_id(str(error), operation, error.response)
+    if message == str(error):
+        return error
+    labelled = type(error).__new__(type(error))
+    labelled.__dict__.update(vars(error))
+    labelled.args = (message,)
+    if str(labelled) != message:
+        return error
+    labelled.__cause__ = error
+    labelled.__dict__["_original_error"] = error
+    return labelled
+
+
+RetryStrategy = Callable[[Exception, int], RetryDecision]
+
+
+def decide_on_original_error(strategy: RetryStrategy) -> RetryStrategy:
+    """Return a retry strategy that calls ``strategy`` with the error that
+    botocore raised.
+
+    The step raises the copy that :func:`add_request_id` made. The core SDK
+    calls the step retry strategy with that copy. The copy holds the error
+    that botocore raised in its ``_original_error`` attribute. The function
+    passes on that error instead: the same object as before the request ID
+    existed, with its message. So every strategy decides as it did before.
+
+    Any other error passes unchanged. This includes a ``ClientError`` without
+    ``_original_error``, and the errors that the MicroVM operation creates
+    itself, such as ``MicrovmStateError``. Their messages have the request
+    ID.
+
+    The core SDK records the step failure before it calls the strategy. So
+    the record, and the handler, keep the message with the request ID.
+
+    Neither error changes its message at any time. So a traceback that code
+    formats during the strategy call shows the same message as at any other
+    time.
+    """
+
+    def decide(error: Exception, attempt: int) -> RetryDecision:
+        original = (
+            getattr(error, "_original_error", None)
+            if isinstance(error, ClientError)
+            else None
+        )
+        return strategy(
+            original if isinstance(original, ClientError) else error, attempt
+        )
+
+    return decide

packages/aws-durable-execution-sdk-python-extras/src/aws_durable_execution_sdk_python_extras/microvm/config.py

@@ -138,6 +138,22 @@ class MicrovmBaseConfig:
     It is the second retry tier. The first tier retries inside one step
     attempt: the boto3 client retries each call, and the request step retries
     the POST. Defaults to :func:`default_microvm_retry_strategy`.
+
+    For a failed Lambda MicroVMs call, the strategy receives the
+    ``ClientError`` that botocore raised, with its class, its ``response``,
+    and its message. The step records a copy whose message ends with the
+    request ID. So a strategy that matches the message, such as
+    ``create_retry_strategy(RetryStrategyConfig(retryable_errors=[...]))``,
+    decides on the message of the service. An error that the operation
+    creates itself, such as "RunMicrovm returned no microvmId", has the
+    request ID in its message when the strategy receives it.
+
+    A ``ResourceNotFoundException`` from ``GetMicrovm``, ``ResumeMicrovm``,
+    or ``CreateMicrovmAuthToken`` is the exception. It means that the
+    MicroVM no longer exists. So the operation raises ``MicrovmStateError``
+    of ``aws_durable_execution_sdk_python_extras.microvm.lifecycle``
+    instead, and the strategy receives that error. Its ``__cause__`` is the
+    ``ClientError``, and its message ends with the request ID.
     """
 
     opener: urllib.request.OpenerDirector | None = None