Repository navigation
test: add LMI cloud lifecycle regressions - #743
zhongkechen wants to merge 13 commits into
Conversation
Add opt-in cloud deployment and lifecycle coverage for Python LMI. Assert invocation deadlines, early branch cleanup, replay, and worker recovery; keep issue 741 regressions red until the SDK fix.
The root only coordinates workspace packages. Disable its implicit editable install so the LMI test directory does not trigger setuptools' multiple top-level package discovery error when starting examples. Exercise examples environment initialization in the unprivileged LMI harness job and verify that it imports the checked-out SDK.
Run the complete cloud matrix on every trusted PR update, including drafts, and every main push without labels or path filters. Preserve existing cloud credential restrictions for forks and Dependabot. Use run-owned resources without a cross-run concurrency group so newer commits cannot replace another commit's pending cloud job.
Initialize explicit hold/release objects before invoking fixtures and report missing or forbidden controls as infrastructure failures. Signal the winner only after the losing step has entered held I/O, and reject cleanup assertions when that loser exited before the winner was ready. Poll events by case with bounded parallel reads, correlate checkpoint holds to their request, and check stale effects before waiting for a retry on a potentially pinned worker. Cover the observed false positives and full fixture handshakes with regression tests.
Keep LMI on Python 3.14 with environment concurrency one and two. Exercise root finally before PENDING, bounded nested progress, concurrent shared-handler invocations, in-flight return/failure cleanup, and late operations from abandoned children using Python public APIs. Document the Java/Python mapping and preserve the newly exposed late-step scope defect as an explicit issue 741 regression.
Collect each case's own executions during teardown and leave full-run history/log collection to the final workflow step. Pace history reads and bound throttle-only retries, including reads through the public cloud runner, without retrying invocation or callback writes. Cover pagination, throttling exhaustion, permission errors, and scoped versus full-run collection with focused tests.
Checkpoint a victim's placement decision before starting fault work. If a runtime environment rotates between anchor and victim admission, replace the pair with a bounded retry. Never retry an admitted lifecycle assertion. Keep the admission decision stable during replay.
Hold a repository-wide concurrency slot through deployment, testing, and cleanup without increasing shared provider capacity. Retain up to 100 pending cloud jobs with queue: max instead of evicting older jobs. Keep the harness parallel and document the queue and repository scope.
Deploy one function per native concurrency setting and reuse it for ordinary and deadline scenarios. Run all 28 cases in one cloud job without changing provider capacity or losing regression assertions. Release controls, stop remaining executions, and observe wrapper exit before reusing each function. Extend fault observation for the shared 60-second timeout and retain a bounded emergency release.
Create the persistent stack once and update the same two functions after previous durable executions finish. Preserve failed stacks, code artifacts, and infrastructure after every test run. Isolate controls and events by run ID, expire only run data, and scope cleanup to the current run without deleting functions or buckets. Remove the infrastructure janitor and verify create/update behavior.
Per-test analysis of LMI run 35918433206Scope: workflow run, attempt 1, PR head Evidence reviewed: job/step results, the two Overall: 28 cloud cases — 16 PASS, 12 assertion FAILURES, 0 ERRORS, 0 SKIPS. All 72 harness tests passed. Both cloud jobs successfully built, deployed/read back their functions, collected final evidence, deleted their temporary resources, and published artifacts. Only the cloud assertion steps failed. The ledgers contain no
Configuration and interpretation
Passing casesEach row passed on both c1 and c2 (eight cases per configuration).
The last two passes do not contradict the bounded-cleanup failures below: they verify that work settles after an explicit release, while the failing early-completion tests require bounded cleanup without that release. Failing casesEarly completion —
|
| Scenario | c1: wrapper return after winner | c2: wrapper return after winner | Effects recorded after winner +5 s (c1 / c2) |
|---|---|---|---|
parallel |
7.108 s | 7.709 s | 7 / 8 |
map |
7.209 s | 7.509 s | 7 / 8 |
nested |
6.908 s | 7.011 s | 6 / 6 |
These measured delays are terminated by test cleanup; they are not measurements of an infinite production hang. The source-level unbounded wait is the shutdown(wait=True) cleanup path described in #741 problem 2. Canceling queued futures does not stop an already-running branch blocked in user I/O. In affected workloads this can retain runtime capacity and allow unwanted downstream work after the winning result is selected.
test_invocation_deadline_isolation_and_capacity_recovery (one failure per configuration)
Scenario / expected: Hold a step across the actual 10-second Lambda invocation timeout, submit recovery demand around the deadline, and recover all configured worker slots in the original environment within deadline +5 s. c2 first establishes a healthy companion in that environment.
Actual: Both cases reached and failed #741: full original-environment worker capacity did not recover. Service history independently records InvocationCompleted with Sandbox.Timedout / Task timed out after 10.00 seconds for each original request. The external ledger still records the original request's effects well beyond that timeout:
| Configuration | Fault marker | Last observed effect after invocation deadline | Effects after deadline +5 s |
|---|---|---|---|
| c1 | deadline-baa254d57873 |
+16.043 s | 39 |
| c2 | deadline-76b8fd56a328 |
+16.233 s | 40 |
The original wrappers eventually raised after driver cleanup. Later execution completion cannot satisfy the earlier recovery bound. This directly supports #741 problem 1: a service timeout alone does not stop the LMI Python work or restore full worker capacity.
Scope limit: The c2 companion was released and completed before the deadline; this run establishes that early healthy execution, not continuous healthy-peer progress throughout timeout recovery. The full-capacity assertion still requires both slots, so a probe using only the already-released healthy slot cannot make recovery pass. The test aborts at the recovery assertion; subsequent in-test timing assertions were not reached. The service timeout and continued effects above were checked independently in the retained evidence.
test_service_timeout_retry_does_not_repeat_completed_step (one failure per configuration)
Scenario / expected: After a real invocation timeout, stale-attempt effects must cease within the cleanup bound, and a service retry of the same durable execution must reuse the completed step rather than repeat its body.
Actual: Both fail first at #741: work continued after invocation deadline + cleanup grace. Original-attempt effects continue until +7.549 s (c1) and +8.065 s (c2) after the deadline. In c2 (deadline-158ca58c11ed), a retry of the same execution enters another process in the same environment at deadline +1.078 s, while the original process continues producing effects. This supplies direct evidence of stale-attempt/retry overlap, within #741 problem 1.
Scope limit: The formal retry-wait and completed-step assertions occur after the failing stale-effect assertion and were not executed. Final evidence nevertheless records one success body and one service StepSucceeded for that completed step in each configuration, together with the later retry. That supplementary observation does not turn these cases into passes or establish exactly-once execution for interrupted at-least-once work. No completed-step duplication was observed in these records.
test_abandoned_child_rejects_late_durable_operation (one failure per configuration)
Scenario / expected: After race-parallel has completed, release its losing branch and attempt a new child.step(name="late-work") from that branch's finally. The new operation should raise OrphanedChildException before executing its body or creating a service operation.
Actual: Both fail at #741: abandoned child executed a late user side effect. The ledger records LATE_ATTEMPT, a late-work body, and LATE_ACCEPTED; service history independently confirms admission after ancestor completion:
| Configuration / marker | race-parallel ContextSucceeded |
late-work StepStarted and StepSucceeded |
|---|---|---|
c1 — late-operation-10904bc0b80a |
21:02:31.849 UTC | 21:02:33.321 UTC |
c2 — late-operation-c04640fcc324 |
21:13:27.221 UTC | 21:13:28.819 UTC |
All times are on 2026-09-23. This reproduces #741 problem 3 on both configurations, extending the earlier concurrency-2 evidence already recorded in that issue. The orphan guard checks the new operation's own ID, but that ID does not inherit the already-completed ancestor's state. This creates a risk of admitting new downstream actions after the enclosing operation has completed, distinct from merely allowing an existing I/O call to continue.
The demonstrated boundary is after ancestor completion. These late bodies execute before their outer wrapper's final return; the records do not establish new work beginning after wrapper return.
What this run establishes, and what remains pending
- All 12 cloud assertion failures in this run correspond to the three defects now tracked in [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741. No additional untracked SDK defect is established by this run. They remain ordinary failures, not
xfailor expected-success results. Passing cases and supplementary observations do not satisfy the portions of failed cases that were never reached. - The earlier run 35904960112 had a c1 capacity-provider provisioning failure; its c1 tests did not execute. That was an infrastructure failure, not an expected SDK regression. In the run analyzed here, both deployments and both cleanups succeeded.
- The tested revision used temporary
normal/deadlinedeployments with different timeouts. Current head7bd33b18c5320effbacb86206c8ab12ed9c8ddc3instead retains two persistent shared functions, runs all 28 cases in one job, uses a common 60-second invocation timeout, isolates data per run, and settles case work before reuse. Those deployment/lifecycle changes are not cloud-validated by the older run above. - At the time of posting, the current persistent-deployment run is queued. Current-head local validation has 93 passing harness tests, successful packaging/Ruff, and all 28 cloud cases collected. Cloud validation must still establish first creation, subsequent updates of the same functions, and case/run isolation with the new layout.
The SDK fixes remain in #741; this PR supplies the regressions and stays Draft.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment has been minimized.
This comment has been minimized.
Keep strict cloud regressions manual while the harness runs on PRs. Pin artifact uploads to the immutable revision used on current main. Invalidate old run metadata before build and deployment preflight, preserving the build manifest and preventing prior-run cleanup.
| if e["phase"] in {"WRAPPER_RETURN", "WRAPPER_RAISE"} | ||
| } | ||
| return entered <= exited |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| except (ClientError, ValueError) as error: | ||
| # Every key must exist before Invoke. Missing/forbidden objects are | ||
| # fixture errors, never an implicit hold or release signal. | ||
| self.emit("CONTROL_ERROR", gate=name, error=str(error)) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| def retry_observed(): | ||
| entries = evidence.select(cloud.for_item(fault), "WRAPPER_ENTER") | ||
| return entries if len({e["request"] for e in entries}) >= 2 else None |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
| entered = {e["request"] for e in select(events, "WRAPPER_ENTER")} | ||
| exited = { | ||
| e["request"] | ||
| for e in events | ||
| if e["phase"] in {"WRAPPER_RETURN", "WRAPPER_RAISE"} | ||
| } | ||
| return entered <= exited |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_sajbrsyx7vbqbtm6awgqxcckd5
[P2] Accept verified worker retirement as settlement evidence. A worker retired at its deadline cannot emit either wrapper-exit event. Although evidence.deadline() explicitly permits retirement, this condition remains false even after recovery establishes full capacity in the original environment. Teardown then fails permanently, and configuration['pending'] prevents subsequent cases from running. Carry independently verified, request/process-correlated retirement evidence into settlement alongside wrapper exits, while rejecting unproven termination. Add coverage for retirement without a final wrapper diagnostic.
| except Exception as error: | ||
| save( | ||
| ARTIFACTS / f"invocations/{marker}.json", | ||
| {"payload": payload, "error": str(error)}, | ||
| ) | ||
| raise CollectionError(f"{marker}: invocation API failed") from error |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_gz6jzhkkxu53lgixrg4alhvbvt
[P2] Track ambiguous invocations before allowing function reuse. If Lambda accepts the invocation but its response is lost, this path saves only the payload and raises before adding anything to self.invocations. Per-case teardown consequently cannot stop or observe that execution; with no other invocations, it skips settlement polling entirely. The next case can reuse a function still occupied by the failed case. Register the marker before invoking, recover its ARN from the ledger during teardown, and block reuse until settlement is established. Add a lost-response teardown test.
| except (ClientError, ValueError) as error: | ||
| # Every key must exist before Invoke. Missing/forbidden objects are | ||
| # fixture errors, never an implicit hold or release signal. | ||
| self.emit("CONTROL_ERROR", gate=name, error=str(error)) | ||
| raise |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_jmxq2wca7rviv63o7gphmdyk5g
[P2] Classify transport failures as fixture errors. Botocore transport exceptions such as ReadTimeoutError bypass this handler. If one terminates the losing branch after WINNER_READY, the gate records IO_INTERRUPTED and IO_EXIT without CONTROL_ERROR. The existing SDK can then finish joining the failed branch within the grace period, and early_completion() accepts it as reclamation. Catch botocore transport exceptions, record them as infrastructure errors, and add a negative control injecting a transport failure after winner readiness.
| def retry_observed(): | ||
| entries = evidence.select(cloud.for_item(fault), "WRAPPER_ENTER") | ||
| return entries if len({e["request"] for e in entries}) >= 2 else None |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_e5ywr3cj6ap55eketmslbmdrga
[P2] Observe retry progress before checking for duplicate bodies. WRAPPER_ENTER precedes execution of any steps. This predicate can succeed before the second invocation reaches success, so the subsequent body count remains one even if the retry later repeats that step. Teardown can also stop it before duplication occurs. Wait for the second request's BLOCKED or EFFECT event from controlled-io, which follows the completed step, before checking the count. Add a negative control delaying retry progress after wrapper entry.
| cloud.release(healthy_gate) | ||
| cloud.finish(healthy) | ||
| evidence.lifecycle(cloud.for_item(healthy)) |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_qccfsm2kuodjvzkpbn7upriark
[P2] Keep a healthy peer active across the fault deadline. The healthy invocation is released and completed before the driver waits for the fault's deadline. This establishes ordinary overlap but cannot detect deadline cleanup that interrupts unrelated active work. Recovery probes need not start before that boundary either. Start the healthy peer sufficiently late to retain its own timeout budget, hold it across the fault deadline, and require continued progress from the same request/process before releasing it. Include a negative control where deadline handling interrupts the peer.
Codex AI reviewFive findings: cleanup gaps at Reviewed commit |
This branch was successfully deployed
1 participant
Adds real-LMI regression coverage for invocation deadlines, branch cleanup, and abandoned-child operations in #741. A persistent deployment runs 28 Python 3.14 cloud cases using two shared functions with native environment concurrency 1 and 2, one execution environment each, and a common 60-second invocation timeout.
Closes #742. This test PR does not fix or close #741. Its regression assertions remain ordinary failures until the SDK/runtime behavior satisfies them; no assertion is weakened, skipped within a requested cloud run, or converted to
xfail.python-lmi-e2estack/bucket and fixed-namepython-lmi-e2e-c1/python-lmi-e2e-c2functions. Verify ownership, artifact identity, native concurrency, durability, scaling, and$LATEST.PUBLISHED; wait for prior executions before updating. Failed stacks remain available for diagnosis, and cleanup retains functions, bucket, and content-addressed code.workflow_dispatchuntil the [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741 contract is met, so known failures do not block every unrelated PR/main build. Keep one repository-wide cloud concurrency slot through deployment, tests, evidence collection, and cleanup. Artifact actions are pinned to the immutable v7.0.2 revision used on current main.build.jsonfor artifact verification. A failed preflight cannot make an exit trap collect/release/stop the preceding run. Archive prior evidence before starting another run in the same checkout.Current head
1fd8900calso includes upstream main04daf408. The incoming change touches only the standalone MicroVM worker and its tests; LMI/core/OTel source trees remain unchanged. That worker tree is identical in both refreshed Python PRs and passed 261 tests, type checking (13 files), and formatting/lint. Current-head CI is tracked separately.Validation of the unchanged LMI implementation at
ab723eff:git diff --checkpass. Actionlint 1.7.12 passes with only its pre-existing unsupportedqueuekey diagnostic excluded.5141c9c) yields 2 passes / 3 failures: expired admission and late orphan rejection pass; non-cooperative worker reclamation remains unresolved.7bd33b1, completed deployment/readback, collection, and cleanup. It recorded 16 cloud passes / 12 failures: 11 [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741 assertions and one concurrency-2 recovery placement failure after probes reached a replacement/unrelated environment. That historical run is not acceptance of the updated head.