Skip to content

ateapi: finish a suspend whose caller went away - #3125

Closed
Aditya Shantanu (aditya-shantanu) wants to merge 1 commit into
agent-substrate:mainfrom
aditya-shantanu:fix/suspend-survives-client-cancel
Closed

Aditya Shantanu (aditya-shantanu) wants to merge 1 commit into
agent-substrate:mainfrom
aditya-shantanu:fix/suspend-survives-client-cancel

Conversation

@aditya-shantanu

Copy link
Copy Markdown
Collaborator

Fixes #2394. Related: #2836, whose trigger no longer occurs because the first suspend finishes and a retry gets Aborted while it holds the lease.

A client that cancelled SuspendActor after MarkSuspending left the actor SUSPENDING for good. The workflow ran on the request context, so it stopped with the caller and released the lease. Nothing else finalizes a SUSPENDING actor, ApplyReportedSnapshotState refuses atelet's report of the committed snapshot while the actor transitions, and the worker slot stayed pinned until a retry. Measured during the pre-GA chaos runs: a Locust restart wedged 4 actors this way, and the retried suspend crashed them.

The lease and the workflow steps now run under a context detached from the caller, bounded by MaxRPCDeadline (the cap the RPC already had, now exported from controlapi so main and the workflow share it). The caller's context is checked once before MarkSuspending, so a caller that is already gone starts nothing. After that the workflow returns what the suspend did and logs the outcome when the caller is no longer there to see it. The lease context ends only when renewal fails or the budget runs out. SIGTERM does not cancel the detached workflow; it runs until the process exits and the lease row lingers up to its TTL, so a retry right after a restart can get Aborted. The actor ends SUSPENDING in that case, as before.

Tests: unit rows cancel before MarkSuspending (Canceled, nothing started), during Checkpoint and before finalize, sync and async (both end SUSPENDED, lease held throughout, one Checkpoint); all fail on main. The e2e TestCancelledSyncSuspendLeavesResumableActor cancels a sync suspend as soon as the actor is SUSPENDING and expects it to end SUSPENDED and resumable whether the client waits or retries; it fails if the suspend finishes before the cancel lands. On main at 4b4fbb2 both rows fail on a GKE gVisor cluster (recorded on #2394); it has not yet been run on a cluster with this change, so the kind e2e lanes here are the first run. go test -race -count=3 -shuffle=on ./cmd/ateapi/..., gofmt, golangci-lint and boilerplate are clean.

AI use: written with Claude Code for the change, tests and an adversarial self-review pass by a second session; I reviewed the final diff and the test results by hand.

A client that cancelled SuspendActor after MarkSuspending left the
actor SUSPENDING for good. The workflow ran on the request context, so
it stopped with the caller and released the lease. Nothing else
finalizes a SUSPENDING actor, ApplyReportedSnapshotState refuses
atelet's report of the committed snapshot while the actor transitions,
and the worker slot stayed pinned until a retry. Before atelet answered
a retried Checkpoint from the snapshot it committed, the retry ran
runsc checkpoint on a sandbox that had already exited and crashed the
actor.

Run the actor lease and the workflow steps under a context detached
from the caller, bounded by MaxRPCDeadline, the cap the RPC already
had. Check the caller's context once before MarkSuspending, so a caller
that is already gone starts nothing. After that the workflow returns
what the suspend did, and logs it when the caller is no longer there
to see it. The lease context now ends only when renewal fails or the
budget runs out.

The e2e test cancels a sync suspend as soon as the actor is SUSPENDING
and expects it to end SUSPENDED and resumable, whether the client waits
or retries.
@dims

Copy link
Copy Markdown
Collaborator

Did you see #2558 and #2907 ?

@aditya-shantanu

Copy link
Copy Markdown
Collaborator Author

I did not. Things are moving pretty fast here. Let me review them now and dedup accordingly.

@aditya-shantanu

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #2558, which I had missed. It detaches every lifecycle workflow from the caller's cancellation in acquireLease, which covers what this PR did for suspend only, and #2602 handles the atelet side of the same cancel.

Three small things from here that could ride along with #2558 (or follow it) if they are useful:

  1. A ctx.Err() check right before MarkSuspending, so a caller that is already gone starts nothing (today the detached workflow would still mint a snapshot for a client that left before anything was committed).
  2. One Info log at the end of the RPC when the caller's context is done, naming the actor and the outcome. With the detached workflow the interceptor logs the caller's view and the lifecycle duration metric records a success the client never saw, so nothing in the logs says the caller left.
  3. The e2e TestCancelledSyncSuspendLeavesResumableActor in internal/e2e/suites/demo on this branch (https://github.com/aditya-shantanu/substrate/blob/fix/suspend-survives-client-cancel/internal/e2e/suites/demo/suspendcancel_test.go): it cancels a sync suspend as soon as SUSPENDING is observed and expects SUSPENDED and a successful resume whether the client waits or retries; it fails on main at 4b4fbb2 in both rows and fails loudly if the suspend finishes before the cancel lands. Happy to open it as a separate PR once ateapi: run lifecycle workflows to their end when the caller goes away #2558 is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A client cancel during SuspendActor lets a retry crash the actor mid-checkpoint

2 participants