Skip to content

feat: Cancel queued and running work items (#280) - #295

Merged
bitbiter-dev merged 4 commits into
masterfrom
feat/280-cancel-work-items
Oct 21, 2026
Merged

bitbiter-dev merged 4 commits into
masterfrom
feat/280-cancel-work-items

Conversation

@bitbiter-dev

@bitbiter-dev bitbiter-dev commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Closes #280. Part of #276 (design: ADR 0008, #276 "Execution" and "Admin API").

Summary

An admin can cancel work:

POST /api/v1/admin/work-items/{id}/cancel     admin only
  Queued   → Cancelled at once                 200 + item
  Running  → CancelRequestedAt = now           202 + item, Location: the item
  finished →                                   409 + item as it is
  unknown  →                                   404

The write is one status-guarded statement in EfWorkItemCanceller (Core), so a cancel racing a claim can't be lost. Either the cancel locks the row first and the claim's SKIP LOCKED passes over it, or it sees the claimed row as Running and records the request:

UPDATE "WorkItems"
SET "Status" = CASE WHEN "Status" = 'Queued' THEN 'Cancelled' ELSE "Status" END,
    "FinishedAt" = CASE WHEN "Status" = 'Queued' THEN now ELSE "FinishedAt" END,
    "CancelRequestedAt" = COALESCE("CancelRequestedAt", now), ...
WHERE "Id" = id AND "Status" IN ('Queued', 'Running')
RETURNING "Status"

On the Worker, the heartbeat's own UPDATE returns whether a cancel was requested:

 RunAsync(work)
   heartbeat every 2 s
-    Lost → cancel handler, stop
+    Lost            → cancel handler, stop
+    CancelRequested → cancel handler, keep beating until it returns
   handler ends
     Lost                                      → write nothing
+    CancelRequested and (returned or OCE)     → CancelAsync: Cancelled, last progress (+ summary if any), fenced by Attempt
     returned                                  → Succeed
     OCE and shutdown                          → Release
     otherwise                                 → Fail

A requested item is never queued again. Release (shutdown) and lease recovery end it Cancelled with no error, even at the stall limit. Otherwise a deploy or a dead Worker would drop the cancel.

The DTO gains cancelRequestedAt and resultIsPartial. A Cancelled item now shows its result, and resultIsPartial is true for it. For today's kinds the handler throws on cancel, so the partial outcome is the last progress snapshot. This was decided with the maintainer, and IWorkHandler is unchanged.

Architecture rule work-items-change-only-through-queue-or-cancel: in the API, Core and Infrastructure, only EfWorkQueue, EfWorkItemReader, EfWorkItemCanceller, AnichronDbContext and the Domain types may depend on the WorkItem entity. The API can't reach the DbContext (api-reaches-data-through-repositories) or the Worker (api-does-not-use-worker), so this closes the remaining path: db.WorkItems, Set<WorkItem>() and any lambda over the entity. Two gaps remain, and a type check can't close either:

  • raw SQL that names the table, which stays a review concern (all of it carries // Stryker disable all and an ADR 0005 row);
  • a write added inside the exempt reader.

Due-ness: a cancelled periodic item counts as that run, so the next one follows a full interval later. If it didn't count, the scheduler would enqueue the kind again at its next poll and undo the cancel. EfWorkItemStoreTests.IsDueAsync_FinishedAfterTheCutoff_IsNotDueAsync already pins this for Cancelled, and it is now written in CLAUDE.md and the CancelRequestedAt column comment. An admin "run now" is the later per-kind enqueue endpoint.

Also in this branch: docs: Leave the unit suite to the pre-push hook. This CLAUDE.md note was requested separately by the maintainer. The hooks were checked first: pre-commit runs format and build, pre-push runs the unit suite.

Evidence

  • Contract tests on real PostgreSQL (WorkItemCancellerTests, WorkItemStoreTests):

    • queued → Cancelled and never claimed;
    • running → request recorded, first request kept;
    • finished (Succeeded, Failed, Cancelled) → Finished, row unchanged; unknown → NotFound;
    • 40 rounds of claim racing cancel: either Cancelled and unclaimed, or claimed and the heartbeat reports CancelRequested;
    • heartbeat reports the request and still extends the lease;
    • CancelAsync keeps progress and result, and is fenced (zombie test);
    • release and recovery of a requested item → Cancelled, even with maxStalls 1.
  • Release/recovery guard: Before: with the CancelRequestedAt == null guards removed, Failed: 3, Passed: 53. After: Passed: 56.

  • Executor (WorkExecutorTests, Timeline):

    • cancel request → handler token cancelled, heartbeats go on;
    • handler stops → CancelAsync with its last progress;
    • handler returns a summary → kept as the partial result;
    • handler fails → Fail;
    • shutdown after the request → CancelAsync, not Release;
    • request then fenced → nothing written.
  • Endpoint: 200/202/409/404, and 404 when the item vanishes before it is read. HTTP tests cover the route and non-admin 403.

  • Architecture rule:

    • Before: the rule fails on a temporary RogueWorkItemWriter in Core that runs db.Set<WorkItem>().ExecuteUpdateAsync. The first version of the rule, which checked only db.WorkItems, would have missed this.
    • After: with the violator removed, the rule passes (Total: 31, Failed: 0).
    Rule_HasNoViolationsBeyondItsExceptions(rule: "work-items-change-only-through-queue-or-cancel") [FAIL]
      ... broken by these types ... found at least these items {"Anichron.Core.Work.RogueWorkItemWriter"}.
    Total: 31, Failed: 1
    
  • Suites: unit 1,566 passed; integration 56 passed; scoped mutation 100.00% (346 mutants tested). EfWorkItemCanceller.CancelAsync has its ADR 0005 row, and mutation-guard.sh, derive-db-docs.sh --check and dotnet format pass.

Merge Danger

Door: one-way (mild)

Migration CancelWorkItems adds the nullable column WorkItems.CancelRequestedAt. Rolling back the code is fine. Rolling back the column needs a down migration.

Blast Radius: work-queue

Every heartbeat is now a raw UPDATE … RETURNING, replacing the EF ExecuteUpdate. The UpdatedAt-only-on-change contract tests still pass. A Worker deployed before the API is harmless: no one can request a cancel yet. An API deployed before the Worker records requests that an old Worker ignores until its items end normally.

🤖 Generated with Claude Code

bitbiter-dev and others added 4 commits October 21, 2026 13:25
POST /api/v1/admin/work-items/{id}/cancel cancels a queued item at once
(200) or records CancelRequestedAt on a running one (202); a finished item
is 409 and an unknown id 404. The write is one status-guarded UPDATE, so a
cancel racing a claim either cancels the queued item or reaches the
running one.

The heartbeat reports the request, the executor cancels the handler and
keeps the lease until it returns, and the item ends Cancelled with its
last progress, shown as partial. Release and lease recovery never requeue
a requested item. Architecture rule api-writes-work-items-only-by-cancel
keeps every other work-item write in the Worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TE (#280)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l due-ness (#280)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bitbiter-dev
bitbiter-dev merged commit 1460389 into master Oct 21, 2026
12 checks passed
@bitbiter-dev
bitbiter-dev deleted the feat/280-cancel-work-items branch October 21, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Cancel queued and running work items

1 participant