test: add LMI cloud lifecycle and concurrency regressions

[zhongkechen]

This adds real Lambda Managed Instances (LMI) regression coverage for the invocation lifecycle and executor problems in #726. The fixtures assert the required behavior: wrapper responses follow root/task cleanup, deadlines bound invocation-owned work, and concurrent executions make progress without interfering with each other. The cloud regressions are expected to fail until the corresponding SDK fixes land.

Pull requests run local fixture and evidence validation on Java 17 without cloud credentials, enforcing the SDK's minimum JDK API compatibility even for fixture-only changes. Real cloud execution is manual-dispatch only.

Issue Link, if available

Closes #727. Regression coverage for #726; this PR does not fix or close #726.

Description

• Deploy the SDK commit under test on Java 25 / arm64 with LMI invocation concurrency 1, 2 and 8. Five persistent fixtures cover isolation, suspension/replay, cooperative deadline cancellation, a non-cooperative child, in-flight success/failure, shared fixed-pool and nested orchestration, and repeated warm executions.
• Observe actual wrapper, root and task boundaries, with one JVM-wide lock covering boundary counter changes, sequence assignment and event snapshots, and a request-local ledger pairing every real checkpoint/poll CALL with its finally EXIT. Lifecycle and drain checks derive request identities from all diagnostics, including activity whose wrapper records have not arrived. Every observed request must have matching wrapper boundaries; activity from a previously unseen retry resets the cleanup quiet period.
• Require overlapping requests from distinct execution markers in the same JVM. Multiple retries of one execution cannot substitute for another execution. Pair suspension cleanup boundaries with the original JVM's first PENDING request, and wait for an admitted healthy peer's heartbeat inside that exact interval; reordered logs, later invocations and replacement JVMs cannot supply its evidence.
• Bind deadline, interruption, task-exit and wrapper-return checks to the original victim request. Prove worker recovery using a target-JVM probe while every original healthy peer's complete held-task interval still covers admission. Require the final full-concurrency check to remain in the requested original JVM, including after anchor replay. Validate every victim retry after cleanup; the stubborn-child allowance permits residual task exit after an exceptional wrapper return but never late SDK work or root exit.
• Establish separate root and child barriers for fixed-pool and nested starvation. Neither child submits nested work before both children enter. Setup and progress have separate budgets; a test-only pool expansion remains a regression failure.
• Create or update the persistent java-lmi-e2e stack, fixed-name functions, log groups and private staging/control bucket using an existing dedicated CAPACITY_PROVIDER_ARN. Each function has one 2 GiB / 1 vCPU environment and uses $LATEST.PUBLISHED. Verify the content-addressed jar, runtime deployment identity, scaling and durable configuration; wait for earlier executions before updating. Retain infrastructure and code artifacts after success or failure, with per-run control objects expiring after one day.
• Give each invocation a stable durable execution name derived from its run ID and marker, preventing CLI retries from creating another execution for the same marker. When all invoke responses are lost, cleanup and collection reconcile acceptance through recorded diagnostics or a bounded, paginated lookup by exact execution name and published qualifier. Recovered ARNs are persisted while the original transport failure remains visible. Final collection merges ARNs from current-run invocation response artifacts with diagnostic-derived ARNs, including executions that failed before fixture logging. Restore cached diagnostics and raw logs before refresh so a failed query for one fixture cannot discard its prior evidence.
• Keep setup, collection and SDK assertion failures distinguishable. Observe every invocation future, remove successfully released control gates from tracking while retaining failed releases for retry, make launched executions terminal, and require quiescence before reusing a fixture. Preserve primary and cleanup failures separately, abort subsequent cases when isolation cannot be restored, and retain histories, diagnostic logs and JUnit reports.
• Confine OIDC and AWS secrets to the serialized manual cloud job, with separate credential sessions for deployment, assertions and final collection. The fixture Maven parent follows the current reactor, with a regression test guarding version alignment.

Fixed-pool retries require affirmative barrier-entry evidence that the two roots landed in different JVMs, followed by terminal FAILED states and runtime quiescence. Delayed or missing barrier logs cannot trigger a retry, and escape evidence from every attempted cohort is checked again before accepting a result. The eight-second JVM progress assertion and fixed-pool observation windows are unchanged.

Healthy peers and anchors must report SUCCEEDED and normal lifecycle ordering on their originally admitted request; eventual durable retry success cannot replace that outcome. Timeout peers and replay/in-flight anchors collect their full evidence before acceptance. Physical cleanup can settle a late root exit without erasing its earlier ordering failure. After release, stop, and quiescence attempts, the driver audits every case invocation Future, including discarded placement attempts: reconciliation still drains accepted executions, but failed, cancelled, or unsettled driver requests remain collection errors. Primary SDK assertions and cleanup failures remain separately reported.

Post-stop cleanup also allows an already running stubborn retry its configured bounded hold (at most 120 seconds), followed by the existing 30-second collection margin and 20-second quiet period. Ordinary cleanup stays at 50 seconds; original invocation deadline + 5s, recovery, and progress acceptance remain unchanged.

Before an otherwise successful case can pass, every launched request is validated against the existing lifecycle contract after cleanup has completed. This includes recovery probes, discarded placement attempts, and retries. Only the already-supported stubborn victim permits residual task exit; primary and cleanup failures are preserved.

The initially pinned healthy admission, timeout-victim request, and warm JVM selections are immutable. Final collection revalidates each selection using its existing predicate; healthy requests must also retain their original SUCCEEDED outcome. Changed evidence fails acceptance without repinning earlier timing or occupancy evidence. The original timeout task identity and each warm environment are rechecked against the complete trace using their existing ordering and environment predicates. Admitted fixed/nested cohorts are revalidated once against final membership, barrier and progress evidence, so a later retry cannot replace an original participant.

See lmi-tests/README.md for prerequisites, budgets, commands and infrastructure ownership. Production SDK behavior and public APIs are unchanged.

Demo/Screenshots

Not applicable. Cloud diagnostics and JUnit results are uploaded as workflow artifacts.

Testing

Current head 37a044979633ec56c06b411c5840136540f85f3f contains upstream main 67bad28cd9ef1b0808bed193f8a15be75ce457ed through parent edeb5075; the follow-up changes only the Python evidence driver, its tests, and README. The PR remains test-only relative to that main; the core lifecycle fixes remain separate.

• python3 -B -m unittest discover -s lmi-tests/tests -v — 179/179 passed, including the real local CLI transport contract and retained lifecycle/recovery controls. The ten fixed-placement controls fail against the pre-correction edeb5075 driver and pass after the correction. Further counterexamples reproduce original healthy-request failure hidden by a successful retry, healthy root exit after wrapper return, and late request failures hidden by successful cleanup on 4f1b7662; the new assertions reject them while preserving healthy and delayed-but-correct evidence controls.
• Six post-stop settlement controls pass, covering a running retry, delayed exit logs, the fixture hold cap, bounded failure for work that never exits, and the unchanged ordinary cleanup allowance. Five fail against the prior 8c05b960 driver; the ordinary-budget control remains green.
• Nine case-level lifecycle controls pass, including delayed valid logs, probe and discarded-attempt violations, the supported stubborn exception, and primary/cleanup failure preservation. Four violations were incorrectly accepted by the preceding 160169fc driver and now fail acceptance.
• Ten delayed-identity/cohort driver controls pass for healthy admission, timeout-victim identity, and warm JVM continuity. Seven negative outcomes fail the prior 8c9292d5 driver; unchanged request identities, delayed same-request evidence, and the single-JVM warm/replay sequence remain valid.
• Unchanged Java source at parent edeb5075: mvn -o -B clean verify on Java 17 passed all ten reactor modules: 2,972 tests passed, 34 conditional/cloud tests skipped, zero failures or errors. This includes the LMI fixture build and Java tests.
• Parent edeb5075 full mvn -o -B spotless:check passed; no Java files changed in this follow-up. Current git diff --check passed.
• Cloud execution was not repeated for this upstream synchronization. Previous real LMI evidence: https://github.com/aws/aws-durable-execution-sdk-java/actions/runs/35794439406/attempts/2. Cloud regressions remain manual-dispatch only and cannot be considered fixed merely because local fixture validation passes.

Checklist

• I have filled out every section of the PR template.
• I have tested this change locally; cloud validation limits are recorded above.
• Production SDK behavior and public APIs are unchanged.
• Cloud credentials are unavailable to pull-request jobs.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[zhongkechen]

Per-test analysis of LMI run 35794439406

Scope: workflow run, attempt 2, PR head 340433bcaa2a10cd2363ff01c60120c819face1e. The artifact manifest records the checked-out PR merge commit as 9f57aeb1919fc75835c1e9abd3d754c560ffc664. This describes that run, not subsequent revisions of this PR.

Evidence reviewed: job results and the lmi-e2e-35794439406-2 artifact, including junit.xml, diagnostics.json, cloudwatch.json, invocation responses, execution histories, and the timeout report. The fixture and assertion code was also checked.

Overall: 13 tests — 3 PASS, 6 assertion FAILURES, 4 collection ERRORS. Build, deployment/readback, final evidence collection, resource deletion, and artifact publication succeeded. The workflow failed in the cloud assertions step.

What the configurations mean

• default1, default2, and default8 use the SDK's default cached executor. Their numbers are the maximum concurrent Lambda invocations per execution environment, not executor thread counts or vCPU counts.
• fixed2 and nested2 each allow two concurrent Lambda invocations and share a fixed two-thread SDK user executor.
• Timeout fixtures use a 60-second invocation timeout and a separate 240-second durable execution timeout. The deliberately blocked tasks have an 80-second test escape. The cleanup assertion allows the invocation deadline plus 5 seconds; recovery admission allows plus 8 seconds.
• A test-only escape is a failure signal. Eventual completion after expanding the executor or releasing an escaped task does not make the regression pass.

Links to the corresponding problems in #726

• Problem 1 — invocation cleanup waits indefinitely: missing invocation deadlines, task cancellation/draining, and the resulting occupied runtime capacity. Its separate interrupted-drain/checkpoint-shutdown subproblem is not demonstrated by this run.
• Problem 2 — PENDING can precede root-handler cleanup: the root future is not included in the required drain before returning.
• Problem 3 — shared fixed-pool starvation: blocking orchestration occupies the same workers needed by its children/steps.

The links below distinguish direct reproduction, downstream effects of earlier failures, and the contract a test intended to check but did not reach. A collection error is not automatically another SDK defect.

Passing cases

Test	Result and demonstrated scope
baseline-concurrency1	PASS. Successful/failed checkpoints, real wait/resume, stable identities and stored failure meaning passed. This case does not contain delayed root cleanup.
default2-isolation	PASS. Two distinct invocations overlapped in one JVM, returned their own markers, and satisfied the tested normal lifecycle boundaries.
default8-isolation	PASS. Eight distinct invocations overlapped in one JVM and returned their own markers.

Non-passing cases

Severity rates potential customer impact in the affected workload; it is not a production-incident classification or a measured customer incidence rate. The tests establish failure mechanisms, not how many customers use the triggering patterns. A High rating for a related SDK issue does not turn a collection error or coverage gap into another confirmed product defect.

default2-suspend-cleanup-replay — assertion failure

• Scenario: On the function configured for two concurrent invocations per JVM and the default cached executor, run a victim through a successful step, a deliberately failed-and-caught step, a durable wait and replay. Keep one healthy companion active in that JVM. The victim delays its root-handler finally by two seconds.
• Expected behavior: Root cleanup and root exit must precede a normal PENDING response. The healthy companion must continue progressing during cleanup. A real resumed invocation must reproduce checkpointed successes/failures without rerunning their bodies or changing operation identity.
• Actual behavior: The SDK wrapper returned normal PENDING while the root's invocation-local finally was still running. For 4d566d3f8b4c-victim-0, wrapper return was sequence 126, root exit was 132, 1.999 seconds later. Replay and the healthy companion's progress checks passed before the ordering assertion failed.
• Severity: Medium generally; potentially High when cleanup protects critical resources or state. Customer impact is conditional on what the delayed cleanup touches and how long it lasts. It can race with later work, but trivial, invocation-private cleanup may have little visible effect. This run confirms the ordering defect; it does not demonstrate resource corruption, repeated checkpointed side effects, or lost checkpoints.
• Cause and corresponding issue: Direct reproduction of #726 problem 2. Suspension can win CompletableFuture.anyOf before the root finishes unwinding, while the root future is absent from the operations drained by close().

default2-timeout-recovery — collection error

• Scenario: Start an interruptible asynchronous step with an 80-second test escape, then return from the root while the step is still blocked. The Lambda invocation timeout is 60 seconds. Stagger one healthy companion into the same JVM, hold its slot, and submit probes for the victim's slot.
• Expected behavior: Establish the healthy companion first, then observe real deadline handling: request cancellation, actual task exit and outer SDK return by the invocation deadline plus five seconds, continued healthy progress, and probe admission in the original JVM by deadline plus eight seconds while the other slot remains held. No test escape should be needed. Log delivery has a separate observation budget and must not be confused with handler entry or task-exit timing.
• Actual behavior: The test aborted inside healthy_peers(), before its cancellation/recovery assertions. Peer 6be2865fcffd-peer-0-0 entered 0.842 seconds after submission and emitted heartbeats, but its entry event took 9.011 seconds to reach CloudWatch, longer than the driver's 8-second observation window. The peer ultimately succeeded. Separately, final victim traces show no interruption, an 80-second escape, and the first wrapper returning about 20.06 seconds after its invocation deadline; the execution later failed with Sandbox.Timedout.
• Severity: Validation failure; no additional customer-defect severity is established by this collection error. Its immediate cause is the short log-observation window, not a slow or absent healthy invocation. The separately observed timeout/cleanup defect is High severity for affected workloads because it can retain capacity after timeout, including with the default executor. This aborted case does not establish the severity or correctness of the full recovery path.
• Cause and corresponding issue: The immediate error is a test-driver observation-window defect. The intended SDK contract is #726 problem 1, but the complete cancellation/healthy-peer/recovery sequence was not assessed. That supplementary evidence supports problem 1 without making this aborted case complete recovery coverage.

non-cooperative-child — assertion failure

• Scenario: On default2, start an asynchronous child that catches and ignores InterruptedException for a bounded 80-second period. Return from the root immediately. At the end of that period the child attempts child.step("late-work", ...). Maintain a healthy companion and recovery probes around the 60-second invocation deadline; the overall durable execution timeout is 240 seconds.
• Expected behavior: The SDK must attempt cancellation and finish its own cleanup within the invocation deadline plus five seconds. If the child has not exited, the outer invocation must take a bounded invocation-level failure path, not claim clean success or suspension. The child may continue its deliberate bounded loop, but further durable work must be rejected before its body/checkpoint is admitted. Healthy progress and timely recovery of the original runtime slot must still be demonstrated; Java is not required to forcibly terminate the uncooperative thread.
• Actual behavior: The child recorded no received interruption. The first SDK wrapper returned 20.164 seconds after the invocation deadline, beyond the +5-second cleanup budget, and still emitted a local SUCCEEDED response marker. A late-work body ran about 20.05 seconds after the deadline; its call subsequently raised UnrecoverableDurableExecutionException. The body ran before wrapper return, so the demonstrated boundary violation is work after the invocation deadline, not proof of work starting after wrapper return. The overall durable execution eventually became TIMED_OUT at 240 seconds. No probe entered the original environment within deadline +8 seconds while the healthy slot was still occupied. The healthy peer's task exited at about +12.799 seconds; the earliest probe entered at +13.382 seconds, before the victim returned at +20.164.
• Severity: High severity for affected workloads. The missing deadline/cancellation handling can retain runtime capacity after timeout and admit additional step bodies, creating availability risk and a risk of unexpected or duplicate downstream actions. Exposure is not limited to custom executors; slow or failing dependencies can make deadline overrun relevant to default-executor users as well. The fixture records marker bodies, so external-data corruption was not demonstrated. Later probes may use the released healthy slot and do not prove victim-slot recovery. This rating concerns the missing cancellation attempt, bounded SDK waiting and late-work guard; it does not require Java to forcibly terminate an interruption-ignoring thread.
• Cause and corresponding issue: Direct evidence for #726 problem 1, plus the missing deadline/cancellation gate addressed by proposal A and proposal B's prohibition on late SDK work. The SDK keeps waiting instead of ending its own cleanup within budget, and a late step is admitted far enough to run its body before the operation fails.

return-with-inflight-step — collection error

• Scenario: First hold a healthy invocation in the default2 JVM. Then launch a victim that starts an asynchronous step doing 1.5 seconds of work, waits until it has started, and returns its marker from the root before that step finishes.
• Expected behavior: The required in-flight step must drain or be cancelled before a clean outer response, within the cleanup budget. The execution must return its own marker, the healthy companion must remain usable, and no new SDK checkpoint/poll activity may continue after the closing boundary.
• Actual behavior: The intended return-with-an-active-1.5-second-step victim was never launched. Its prerequisite healthy anchor, 8fe12f1c76a5-healthy, entered 94.167 seconds after submission, beyond the 25-second admission observation budget, then succeeded. At submission, both runtime slots in that JVM were occupied by earlier timeout and stubborn invocations.
• Severity: High severity for the observed downstream availability problem, not an independently confirmed return-path defect. Earlier stuck invocations delayed a healthy request by 94.167 seconds before handler entry. Similar delays could violate customer latency objectives and cause retries in clients with shorter timeouts. The intended return-with-in-flight-work handler was never launched, so that specific cleanup path remains unassessed.
• Cause and corresponding issue: This is downstream capacity interference from the earlier #726 problem 1 cleanup/deadline failures, combined with lack of quiescence between fault cases. It is not a direct reproduction of the target return/drain path because that root handler never started.

failure-with-inflight-step — collection error

• Scenario: First establish a healthy companion in the default2 JVM. Then launch a victim that starts a 1.5-second asynchronous step and deliberately throws from its root while that step is still active.
• Expected behavior: The intended root failure must produce the expected failed durable outcome. Invocation-owned work must drain or cancel within budget, required checkpoint/poll cleanup must still be attempted, and the healthy invocation must remain usable. The failure path must not admit later SDK work after the invocation closes.
• Actual behavior: The deliberately failing root and its asynchronous step were never launched. Healthy anchor e287198ab7b1-healthy entered 73.234 seconds after submission and ultimately succeeded. The same earlier timeout/stubborn invocations occupied both runtime slots when it was submitted.
• Severity: High severity for the observed downstream availability problem, not an independently confirmed failure-path defect. A healthy request was delayed by 73.234 seconds before entry while earlier fault invocations occupied the environment. The intended failing root was never launched; cleanup after a root throws, preservation of its error, and interrupted-drain checkpoint shutdown cannot be rated as separately demonstrated defects from this case.
• Cause and corresponding issue: Again, an indirect consequence of #726 problem 1, with cross-case interference preventing the intended failure-cleanup test from being established.

warm-repeated-batches — assertion failure

• Scenario: Use the warm default2 environment for three batches of a successful execution, an intentionally failed execution, and a wait/replay execution. Capture invocation-owned task/root counts, executor queue depth and JVM thread counts after the executions.
• Expected behavior: After earlier fault work has reached bounded quiescence or has been accounted for separately, each completed warm execution must leave its invocation-owned tasks and roots at baseline, with no accumulated queued work. The final JVM thread count must remain within the test's +16-thread tolerance, allowing cached-pool idle retention. Replay must preserve checkpointed results and failures. The baseline must not mistakenly attribute another case's still-running task to the warm loop.
• Actual behavior: Failing snapshots contain liveTasks=1, liveRoots=0, queued=0. Matching task enter/exit events by JVM sequence attributes the outstanding task to stubborn-child from earlier execution marker 4ec47272176b-victim. It was not created by the warm batch. The thread-growth assertion occurs after this failing check and was not executed.
• Severity: No independently established warm-loop leak severity. The related cleanup/deadline problem is High severity, but this snapshot attributes work from an earlier stubborn child to the warm case. It does not show that the warm loop leaks its own tasks, accumulates queued work, or grows the thread pool. A deliberately non-cooperative child may remain alive for its bounded escape period even with correct bounded SDK cleanup. Establish quiescence or an invocation-specific baseline before assigning a severity to a separate warm-loop resource leak.
• Cause and corresponding issue: This is residual activity associated with the earlier #726 problem 1 scenario, exposed by a global counter without an isolated warm-test baseline. Proposal A's invocation-owned task accounting is relevant to attributing the work correctly.

default8-suspend-cleanup-replay — assertion failure

• Scenario: Repeat the successful-step/failed-step/wait/replay victim with a two-second root finally delay on the function configured for up to eight concurrent invocations per JVM. This cleanup scenario uses one healthy companion in the same JVM.
• Expected behavior: The root and its cleanup must exit before normal PENDING return, while the companion continues progressing. Replay must use stored successes/failures and stable operation identities. Full eight-invocation overlap is established separately by default8-isolation; this scenario tests the cleanup contract on that configuration.
• Actual behavior: For def38ad21146-victim-0, normal PENDING was returned at sequence 267, while root exit was sequence 273, 2.000 seconds later. The successful/failed-step replay sequence and healthy companion progress passed before the ordering failure.
• Severity: Medium generally; potentially High when cleanup protects critical resources or state. The same response/cleanup ordering defect is reproduced on the concurrency-8 configuration. Customer-visible harm depends on cleanup duration and resource ownership. This case uses one healthy companion and does not demonstrate eight simultaneous cleanup races or actual cross-execution data corruption.
• Cause and corresponding issue: A second direct reproduction of #726 problem 2, with the function configured for up to eight concurrent invocations per environment.

default8-timeout-recovery — collection error

• Scenario: Run the 80-second interruptible asynchronous-step victim on the concurrency-8 function with a 60-second invocation timeout. Admit seven healthy companions into the same JVM and keep their slots occupied while probing recovery of the affected slot.
• Expected behavior: All seven companions must be established before interpreting the timeout assertions. The SDK must cancel the victim, bound actual task exit and its outer return by deadline plus five seconds, preserve healthy progress, and restore admission in the original environment by deadline plus eight seconds. Ultimately all configured slots must be usable again. Delayed log ingestion must not itself be interpreted as an invocation that never entered.
• Actual behavior: All seven healthy peers entered approximately 1.809–2.087 seconds after submission and produced heartbeats, but their entry events reached CloudWatch 8.744–9.022 seconds later. The driver failed its 8-second peer-observation window and never reached the formal cancellation/recovery assertions. The peers later succeeded; the victim was still RUNNING in the collected durable-execution snapshot.
• Severity: Validation failure; no additional customer-defect severity is established by this collection error. The peers ran and succeeded; the driver failed to observe their logs in time. The targeted timeout/cleanup problem is High severity in affected workloads, but this case did not reach its cancellation/recovery assertions. Neither the log-ingestion delay nor the victim's RUNNING durable status establishes a separate concurrency-8 availability defect.
• Cause and corresponding issue: Immediate cause: the same log-observation defect as the concurrency-2 case. The test targets #726 problem 1, but this run does not complete that contract's validation at concurrency 8.

fixed-two-roots — assertion failure

• Scenario: Use two concurrent Lambda invocations sharing a fixed two-thread SDK executor. Synchronize both root handlers at a barrier, then have each synchronously wait for its own step. An eight-second test-only escape can expand the pool so the fixture cannot hang indefinitely.
• Expected behavior: Under the executor progress contract asserted by this test, both steps and executions must complete within the eight-second progress budget using the original two-thread executor. Completion must not depend on expanding the pool or replacing the environment; using the escape is a regression failure.
• Actual behavior: Two synchronized roots occupy both workers of the shared fixed executor and synchronously wait for their steps. The pre-escape snapshot is 2 active workers, 2 queued tasks, pool size 2. Neither queued step can run. After eight seconds, the test-only escape increases executor capacity and the executions finish; the use of that escape is the failing assertion.
• Severity: High severity for affected custom bounded-executor configurations. Once blocking roots occupy the available pool, requests cannot make business progress and subsequent work sharing that executor can also stall. The exposure is narrower than the default-executor lifecycle issues: the default cached executor avoids this particular fixed-capacity deadlock. Success after the test expands the pool does not reduce the severity under the original configuration; customer adoption of affected executors is not yet measured.
• Cause and corresponding issue: Direct reproduction of #726 problem 3. Blocking roots and the work they await consume the same finite executor capacity. Proposal C addresses that dependency cycle.

fixed-nested-map-parallel — assertion failure

• Scenario: Use the same two-invocation/shared-two-thread setup. After each root's first ordinary success step, run child-context orchestration containing map and parallel branches, whose handlers also await step work. Retain the eight-second escape for test termination.
• Expected behavior: The roots, first steps, child contexts, map/parallel coordinators and branch steps must progress under the original executor contract within the budget, without an escape. To independently demonstrate nested starvation, the nested contention point must be reached while that original capacity is still in effect.
• Actual behavior: The first ordinary success steps already stall behind the two root handlers: 2 active workers, 2 queued tasks, pool size 2. Escape events occur at sequences 10/12. The nested child bodies only start at 25/26, after executor capacity has been increased.
• Severity: High severity for the demonstrated root-stage deadlock; independent nested-starvation severity remains unestablished. Requests cannot reach their nested workflow under the original bounded executor. Child/map/parallel work starts only after the escape changes capacity, so this run cannot establish that nested orchestration is safe or independently defective after a root-only fix. The confirmed High rating maps to the same bounded-executor availability defect as fixed-two-roots, not an additional demonstrated nested defect.
• Cause and corresponding issue: The observed failure directly maps to the root-starvation part of #726 problem 3. The further nested-orchestration concern in proposal C is relevant but not independently established by this run.

Evidence limits that apply across these cases

• The concrete task/body delays are bounded observations from the fixtures, not proof that a particular production thread ran forever.
• No case here establishes the separate problem-1 subclaim that interruption during task draining skips checkpoint/poll shutdown.
• late-work ran after the invocation deadline; its later rejection does not prove an SDK admission guard worked. These records do not show a new operation starting after the SDK wrapper's recorded return, nor do they prove whether an already-issued remote checkpoint committed after that boundary.
• The passing replay/isolation checks provide no observed completed-step duplication or cross-execution result corruption. Potential external side effects and user-facing latency consequences above are distinguished from the marker-based effects and timings actually measured.

Implications for the PR

1. The red result is meaningful, but not every red case is a validated SDK regression. The two delayed-cleanup cases establish early PENDING; the fixed-pool cases establish root/step starvation; the non-cooperative case supplies deadline, cancellation-attempt, and late-body evidence.
2. Separate log-delivery deadlines from invocation-lifecycle deadlines. The two peer-admission errors are explained by measured CloudWatch ingestion latency. Extend or otherwise improve evidence observation while keeping the actual in-JVM cleanup/recovery deadlines unchanged.
3. Isolate cases after timeout faults. Earlier executions continued/reinvoked and occupied both slots when the later in-flight cases were submitted. The intended victims were never launched. Establish bounded quiescence of previous executions/tasks before interpreting those cases or the warm-loop resource checks.
4. Give nested starvation an independent oracle. Nested child/map/parallel work must be exercised under the original executor contract before the escape can alter capacity; the current root step fails first.

These results support several findings in #726, but the four collection errors, cross-case-contaminated warm check, and nested-coverage limitation still need to be resolved before treating this run as complete acceptance evidence for #727.

[github-actions]

Codex AI review · Finding arf_v1_ga5pssipf2klpgyanljcalpycl

P2 — Revalidate cases after collecting additional evidence

This refresh runs after execute_case() performs its final lifecycle audit. If it discovers checkpoint/poll activity after an earlier request's wrapper return, it merely caches that evidence; the case retains PASS. Subsequent iterations and collect() also never revalidate completed cases. Retain each case's invocation range and re-audit affected cases after the final refresh before accepting their results. Add a driver test that delivers the violating backend-call records during this refresh.

[github-actions]

Codex AI review · Finding arf_v1_dbswwyuldnjprpdvirdkmoem7g

P2 — Recheck replay assertions after cleanup collects logs

If a duplicate BODY record from replay arrives during cleanup, this assertion has already passed. Balanced task boundaries allow cleanup to succeed, and assert_case_request_lifecycles() accepts the duplicate body when it occurred before wrapper return. The suite can therefore pass despite rerunning checkpointed user code. Preserve the replay marker/history and rerun assert_replay() against the final collected trace, including warm replay cases. Add a delayed duplicate-body regression.

[github-actions]

Codex AI review · Finding arf_v1_dtc3qzxfcmprfubu64mxcbpget

P2 — Order warm snapshots by JVM sequence

events_for() preserves first-observed insertion order, so these endpoints need not be chronological. If the final replay's successful-return snapshot arrives before its earlier PENDING snapshot, snapshots[-1] selects the earlier thread count. A final count of 50 can consequently pass against a baseline of 20 when the delayed PENDING snapshot also reports 20. Sort the same-JVM snapshots by sequence before comparing endpoints, and test reversed delivery of the final replay's snapshots.

[github-actions]

Codex AI review

Three P2 findings in lmi-tests/cloud_suite.py: unchecked final evidence (:959), stale replay validation (:354), and unordered warm snapshots (:832). Static review only; tests and cloud execution were not run.

Reviewed commit 37a044979633ec56c06b411c5840136540f85f3f. Workflow run