[Epic]: Durable Functions on LMI #798

Summary

Cross-SDK tracker for reliable Durable Functions execution and observability on Lambda Managed Instances (LMI). This issue brings together invocation deadlines, task cleanup, work admission, Java executor ownership, OpenTelemetry isolation, runtime trace-header propagation, and Workflow Insight export isolation.

The objective is to keep work and telemetry attributable to the correct invocation, preserve durable checkpoint/replay semantics, and validate recovery and progress under real LMI concurrency. Reproductions and affected SDK revisions are recorded in the linked issues; the implementation and test PRs track their remediation.

Why this matters on LMI

LMI execution environments remain active between invocations, and an invocation timeout does not forcibly terminate the function's code. Old work can continue consuming capacity or producing effects while another invocation or retry runs. See the LMI execution-environment lifecycle.

Runtime sharing boundaries differ:

• JavaScript/TypeScript: asynchronous invocations can overlap inside a Node.js worker. Delayed continuations, checkpoint callbacks, and shared plugin state must remain associated with their owning invocation.
• Java: runtime threads share a JVM and handler object. Root/task lifetime, thread-local scopes, and shared executor capacity require explicit coordination.
• Python: the LMI runtime uses separate worker processes, each serving one request at a time. Missing deadline handling or unbounded cleanup can still retain a worker and reduce capacity.

Native invocation concurrency is separate from SDK map/parallel concurrency. An invocation is one handler attempt; a durable execution may span multiple invocations through suspension and replay. Parent completion, root completion, SDK wrapper return, PENDING, and platform timeout are distinct boundaries. Setting native concurrency to one does not address missing deadline handling.

Known problems and issue coverage

The Python orphan-admission entry overlaps the third finding in #741; #641 provides additional existing tracking rather than a separate LMI-only failure. Likewise, the core lifecycle findings can share a cause and should not be counted as independent incidents merely because multiple tests fail.

Implementation and cloud-validation work

The Java stack is #728 → #731 → #733 → #735. Follow each PR's current base, review, and validation status when integrating it. Regression-test PRs provide infrastructure and assertions; they do not themselves establish that the corresponding runtime defect is fixed. Local test results and deployed LMI acceptance must be recorded separately.

Behavioral and compatibility requirements

Core lifecycle

• Prevent new invocation-owned SDK work after the relevant deadline, completed-parent, or disposal boundary. Cancellation and cleanup must not stop a healthy invocation or shut down a caller-owned shared executor/client.
• Distinguish logical cancellation from physical task exit. Define behavior for cooperative tasks, non-cooperative user code/I/O, already-running synchronous service calls, blocked hooks, and custom executors that execute inline. Do not assume that JavaScript, Python, or Java can forcibly stop arbitrary user work.
• Keep checkpoint processing available long enough to settle accepted work. Preserve a terminal result already acknowledged by the service when a later cleanup or metadata-pagination deadline is reached.
• Preserve stored successes and failures, stable operation identity, and replay behavior. A completed step body must not run again on replay; retries of interrupted, uncheckpointed work still follow the configured step semantics.
• Preserve the language-specific suspension model. Java root cleanup ordering must not be implemented in JS by artificially unwinding a business stack suspended across a durable wait.
• Treat Python #783 as a partial cooperative implementation: non-cooperative worker reclamation and fully bounded wrapper shutdown remain part of chore: release version 2.3.0 #741. Treat Java feat: add per-invocation executor isolation #735 as executor isolation, not a guarantee that a fixed-size pool supports arbitrary synchronous nesting.

Observability

• Give each JS/Java invocation its own plugin state, including execution identity, span maps, and scopes. Preserve shared provider/exporter ownership. Validate matching cleanup on success, suspension, retry, and failure, including late callbacks and overlapping executions with identical operation IDs.
• Preserve same-thread scope cleanup where required, correct End ordering, and the original failure when cleanup also fails. Validate exporter failures and flush budgets without claiming durable export acknowledgement or guaranteed delivery after timeout.
• Capture the X-Ray header at invocation entry: context.xRayTraceId on Node.js LMI and Context.getXrayTraceId() on Java LMI. Capture Java's local value before handing work to another thread.
• Carry the header as immutable invocation-local metadata. A runtime-provided carrier is authoritative even when empty or malformed; ordinary Lambda fallback applies only when the carrier is unavailable. Do not copy it into shared environment variables or system properties.
• Test existing custom extractors, older runtime/context implementations, and supported core/plugin combinations. Document migration requirements for the factory APIs and the Java InvocationInfo record-shape change; retaining constructors does not imply unchanged record-pattern source or reflection/value semantics.
• For Workflow Insight, prevent records from different execution ARNs from replacing one another. Scope burst coalescing explicitly and deliver terminal records according to the drain contract in #916.

Acceptance criteria

• Resolve the linked defects or explicitly agree the remaining supported-behavior limits and follow-up work. Record the SDK versions and configuration to which each conclusion applies.
• Complete implementation review, integration with the intended upstream/base, and required checks. Preserve negative controls and defect assertions rather than weakening an unfixed baseline into a pass.
• Exercise the real public SDK APIs through initial invocation, timeout, suspension/resume, retry, early branch completion, and normal/failing return with work in flight. Verify stored outcomes and expected side-effect counts.
• Validate deadline/disposal/parent admission, cancellation isolation, actual task exit, interrupted cleanup, and preservation of acknowledged terminal outcomes. Include bounded fault fixtures for non-cooperative work.
• Validate Java root cleanup and End ordering, concurrent roots, shared-executor ownership, invocation-owned executor shutdown, and the documented nested-capacity limits.
• Validate OTel isolation in both views and Workflow Insight record isolation under overlapping invocations, including cleanup and export failures.
• Validate X-Ray Root, Parent, and Sampled during initial invocation and replay/resume, with sampled/unsampled, unavailable, empty, malformed, and valid headers; ordinary Lambda fallback; and worker-thread handoff.
• Run real LMI cloud tests with the intended SDK artifacts and native concurrency settings. Record deployed commit/artifact identity, configuration, environment/worker/process identity, invocation IDs, lifecycle events, JUnit, logs, and service history. Parallel requests alone do not prove same-environment overlap or recovery of the affected worker capacity.
• Isolate fault cases and retain their evidence. Test-only releases or emergency escapes must not satisfy the preceding recovery assertion. Report provisioning, placement, and collection failures separately from SDK failures; distinguish an attempted checkpoint request from service acceptance. Agree test timing bounds explicitly rather than presenting them as product SLAs.
• Complete the required compatibility/migration guidance, release validation, and release notes. Link final acceptance evidence and released versions from the relevant issues before considering this tracker complete.

Public runtime references

• LMI execution environment and lifecycle
• Node.js runtime on LMI
• Python runtime on LMI
• Java runtime on LMI