Repository navigation
feat: Cancel queued and running work items (#280) - #295
Merged
Merged
Conversation
POST /api/v1/admin/work-items/{id}/cancel cancels a queued item at once
(200) or records CancelRequestedAt on a running one (202); a finished item
is 409 and an unknown id 404. The write is one status-guarded UPDATE, so a
cancel racing a claim either cancels the queued item or reaches the
running one.
The heartbeat reports the request, the executor cancels the handler and
keeps the lease until it returns, and the item ends Cancelled with its
last progress, shown as partial. Release and lease recovery never requeue
a requested item. Architecture rule api-writes-work-items-only-by-cancel
keeps every other work-item write in the Worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TE (#280) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l due-ness (#280) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #280. Part of #276 (design: ADR 0008, #276 "Execution" and "Admin API").
Summary
An admin can cancel work:
The write is one status-guarded statement in
EfWorkItemCanceller(Core), so a cancel racing a claim can't be lost. Either the cancel locks the row first and the claim'sSKIP LOCKEDpasses over it, or it sees the claimed row asRunningand records the request:On the Worker, the heartbeat's own
UPDATEreturns whether a cancel was requested:A requested item is never queued again. Release (shutdown) and lease recovery end it
Cancelledwith no error, even at the stall limit. Otherwise a deploy or a dead Worker would drop the cancel.The DTO gains
cancelRequestedAtandresultIsPartial. ACancelleditem now shows its result, andresultIsPartialis true for it. For today's kinds the handler throws on cancel, so the partial outcome is the last progress snapshot. This was decided with the maintainer, andIWorkHandleris unchanged.Architecture rule
work-items-change-only-through-queue-or-cancel: in the API, Core and Infrastructure, onlyEfWorkQueue,EfWorkItemReader,EfWorkItemCanceller,AnichronDbContextand the Domain types may depend on theWorkItementity. The API can't reach the DbContext (api-reaches-data-through-repositories) or the Worker (api-does-not-use-worker), so this closes the remaining path:db.WorkItems,Set<WorkItem>()and any lambda over the entity. Two gaps remain, and a type check can't close either:// Stryker disable alland an ADR 0005 row);Due-ness: a cancelled periodic item counts as that run, so the next one follows a full interval later. If it didn't count, the scheduler would enqueue the kind again at its next poll and undo the cancel.
EfWorkItemStoreTests.IsDueAsync_FinishedAfterTheCutoff_IsNotDueAsyncalready pins this forCancelled, and it is now written in CLAUDE.md and theCancelRequestedAtcolumn comment. An admin "run now" is the later per-kind enqueue endpoint.Also in this branch:
docs: Leave the unit suite to the pre-push hook. This CLAUDE.md note was requested separately by the maintainer. The hooks were checked first: pre-commit runs format and build, pre-push runs the unit suite.Evidence
Contract tests on real PostgreSQL (
WorkItemCancellerTests,WorkItemStoreTests):Cancelledand never claimed;Finished, row unchanged; unknown →NotFound;Cancelledand unclaimed, or claimed and the heartbeat reportsCancelRequested;CancelAsynckeeps progress and result, and is fenced (zombie test);Cancelled, even withmaxStalls1.Release/recovery guard: Before: with the
CancelRequestedAt == nullguards removed,Failed: 3, Passed: 53. After:Passed: 56.Executor (
WorkExecutorTests,Timeline):CancelAsyncwith its last progress;Fail;CancelAsync, notRelease;Endpoint: 200/202/409/404, and 404 when the item vanishes before it is read. HTTP tests cover the route and non-admin
403.Architecture rule:
RogueWorkItemWriterin Core that runsdb.Set<WorkItem>().ExecuteUpdateAsync. The first version of the rule, which checked onlydb.WorkItems, would have missed this.Total: 31, Failed: 0).Suites: unit 1,566 passed; integration 56 passed; scoped mutation 100.00% (346 mutants tested).
EfWorkItemCanceller.CancelAsynchas its ADR 0005 row, andmutation-guard.sh,derive-db-docs.sh --checkanddotnet formatpass.Merge Danger
Door: one-way (mild)
Migration
CancelWorkItemsadds the nullable columnWorkItems.CancelRequestedAt. Rolling back the code is fine. Rolling back the column needs a down migration.Blast Radius: work-queue
Every heartbeat is now a raw
UPDATE … RETURNING, replacing the EFExecuteUpdate. TheUpdatedAt-only-on-change contract tests still pass. A Worker deployed before the API is harmless: no one can request a cancel yet. An API deployed before the Worker records requests that an old Worker ignores until its items end normally.🤖 Generated with Claude Code