Repository navigation
ateapi: finish a suspend whose caller went away - #3125
Closed
Aditya Shantanu (aditya-shantanu) wants to merge 1 commit into
Closed
Aditya Shantanu (aditya-shantanu) wants to merge 1 commit into
Aditya Shantanu (aditya-shantanu) wants to merge 1 commit into
Conversation
A client that cancelled SuspendActor after MarkSuspending left the actor SUSPENDING for good. The workflow ran on the request context, so it stopped with the caller and released the lease. Nothing else finalizes a SUSPENDING actor, ApplyReportedSnapshotState refuses atelet's report of the committed snapshot while the actor transitions, and the worker slot stayed pinned until a retry. Before atelet answered a retried Checkpoint from the snapshot it committed, the retry ran runsc checkpoint on a sandbox that had already exited and crashed the actor. Run the actor lease and the workflow steps under a context detached from the caller, bounded by MaxRPCDeadline, the cap the RPC already had. Check the caller's context once before MarkSuspending, so a caller that is already gone starts nothing. After that the workflow returns what the suspend did, and logs it when the caller is no longer there to see it. The lease context now ends only when renewal fails or the budget runs out. The e2e test cancels a sync suspend as soon as the actor is SUSPENDING and expects it to end SUSPENDED and resumable, whether the client waits or retries.
Collaborator
Collaborator
Author
|
I did not. Things are moving pretty fast here. Let me review them now and dedup accordingly. |
Collaborator
Author
|
Closing in favor of #2558, which I had missed. It detaches every lifecycle workflow from the caller's cancellation in Three small things from here that could ride along with #2558 (or follow it) if they are useful:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2394. Related: #2836, whose trigger no longer occurs because the first suspend finishes and a retry gets Aborted while it holds the lease.
A client that cancelled
SuspendActorafterMarkSuspendingleft the actor SUSPENDING for good. The workflow ran on the request context, so it stopped with the caller and released the lease. Nothing else finalizes a SUSPENDING actor,ApplyReportedSnapshotStaterefuses atelet's report of the committed snapshot while the actor transitions, and the worker slot stayed pinned until a retry. Measured during the pre-GA chaos runs: a Locust restart wedged 4 actors this way, and the retried suspend crashed them.The lease and the workflow steps now run under a context detached from the caller, bounded by
MaxRPCDeadline(the cap the RPC already had, now exported from controlapi so main and the workflow share it). The caller's context is checked once beforeMarkSuspending, so a caller that is already gone starts nothing. After that the workflow returns what the suspend did and logs the outcome when the caller is no longer there to see it. The lease context ends only when renewal fails or the budget runs out. SIGTERM does not cancel the detached workflow; it runs until the process exits and the lease row lingers up to its TTL, so a retry right after a restart can get Aborted. The actor ends SUSPENDING in that case, as before.Tests: unit rows cancel before
MarkSuspending(Canceled, nothing started), during Checkpoint and before finalize, sync and async (both end SUSPENDED, lease held throughout, one Checkpoint); all fail on main. The e2eTestCancelledSyncSuspendLeavesResumableActorcancels a sync suspend as soon as the actor is SUSPENDING and expects it to end SUSPENDED and resumable whether the client waits or retries; it fails if the suspend finishes before the cancel lands. On main at 4b4fbb2 both rows fail on a GKE gVisor cluster (recorded on #2394); it has not yet been run on a cluster with this change, so the kind e2e lanes here are the first run.go test -race -count=3 -shuffle=on ./cmd/ateapi/..., gofmt, golangci-lint and boilerplate are clean.AI use: written with Claude Code for the change, tests and an adversarial self-review pass by a second session; I reviewed the final diff and the test results by hand.