fix(mothership): settle chat runs no controller owns by waleedlatif1 · Pull Request #8457 · simstudioai/sim

waleedlatif1 · 2026-09-30T08:54:41Z

Summary

A chat run could stay active or paused_waiting_for_tool forever if it never reached a terminal status. That happened when its controller died (deploy, OOM), was superseded with no successor, or was stopped when no controller remained. Nothing ever settled these runs. Production had a large backlog of them, almost all from before the controller-lease protocol. Some chats kept pointing at a dead run, so they could show as busy.

cleanup-stale-executions now sweeps runs that no controller owns:
- Leased runs are settled when their stream holds no chat lock, their replay buffer has expired, and they have been idle for more than an hour. Liveness comes only from the heartbeated chat lock, which a live controller renews every 20 s for its whole life, so a long-running turn is never touched.
- Pre-lease runs (tool_execution_version < 2, no lease) are settled after 24 h idle. They keep updated_at (so the retention clock isn't reset), get completedAt = updated_at (so they don't all land on one day), and carry their own filterable error text.
- Current headless runs are never swept; their own lifecycle settles them.
Stopping a run that no controller holds now settles it as cancelled, instead of leaving it active. cancelled requires a recorded Stop, checked inside the same UPDATE, so a run superseded by a newer turn is never mislabelled.
Every settle is atomic against the controller finalizing or a successor claiming the run: status and owner token are re-checked in one UPDATE. Chats are locked before runs, in the claim path's order, so the two can't deadlock.
Chat markers are cleared in one statement per batch. A cron run handles at most 5k rows, in batches of 500, with a short pause between batches.
Slack-search Assistant runs, which own their run row, never recorded success or a stop: only failures were written, so a successful run stayed active. They now settle complete, cancelled or error once, through the shared run-status helper.

This backlog is the only backfill; no separate one is needed. The sweep drains it gradually after deploy.

Testing

New orphaned-runs.integration.ts runs against real PostgreSQL and Redis:
- settle and skip cases: lock held, recovering controller, replay buffer present, fresh run, current headless run, legacy threshold, and never cancelling an unstopped run;
- sweep and Stop racing controller finalize and successor claim;
- a claim-vs-settle deadlock stress test.
New slack-search/assistant.integration.ts (real PostgreSQL): completed, stopped and failed Slack-search turns settle complete, cancelled and error.
Each guard was reverted and its test went red (Stop-row check, unfinished-status check, owner-token check, lock order).
bun run lint, bun run type-check, bun run check:audits, unit tests for the cron route and lib/mothership/request, and the replay-budget integration suite all pass.

A run whose process died before finalize, whose controller was superseded with no successor, or that was stopped while no controller existed stayed unfinished forever: its chat marker kept pointing at it, so the chat read as busy and a reconnect polled a run nothing would ever end. - Stop now settles the run as cancelled once no controller of its stream holds the chat lock, including after it force-releases a controller that did not exit in time. - The stale-execution cron settles leased runs whose stream holds no chat lock, has no replay buffer left, and has been idle past the orchestration budget (so a reconnect has nothing left to resume), and runs without a lease once idle for 24 hours. A run Stop already closed settles as cancelled, any other as error; its chat marker is released. - Each settle is one conditional update on the run row that requires it to be unfinished, idle, and still naming the controller that was observed, so a finalizing controller or a successor's claim wins or loses against it atomically and the run settles exactly once.

…bel them precisely - Settling now locks the affected chat rows first, in id order, as a controller's claim does. Locking the run and then the chat deadlocked against a concurrent reconnect claim. - Each sweep batch fails on its own, the chat markers of a batch clear in one statement, and a sweep settles at most 5k rows with a short pause between full batches. - A run settles as cancelled only when its user pressed Stop. A newer turn also closes tool admission on older runs, and those now settle as errors. - Runs without a controller lease keep their last write as their completion and retention time and read "never finalized (no controller lease)". - The leased-run grace no longer derives from the orchestration deadline. Liveness comes only from the heartbeat-renewed chat lock; the grace and the replay TTL only bound how long a reconnect can resume a dead run.

A headless turn has no chat lease and no heartbeat, so its age says nothing about whether it is still running once runs have no deadline. The sweep's lease-less rule now applies only to runs admitted before the current tool-execution protocol: every run the current code admits records the current version, so after a deploy no such row can be live. A current headless run is left to its own lifecycle, which always settles it. The protocol version moves beside the other async-run constants so the sweep can read it without importing the repository.

…n settle - Settling a stopped run now passes the Stop-row check as the update's own guard, so it cannot cancel a run nobody stopped; the separate stopped branch is gone and every settle derives cancelled from the Stop row. - Chat lock ownership is read through getChatStreamLockOwners and trusted only when verified, instead of a second Redis read of the same keys. - Settle transactions use the shared DbTransaction type.

vercel · 2026-09-30T08:54:44Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 9:43am UTC

greptile-apps · 2026-09-30T09:00:01Z

Confidence Score: 5/5

[Critical risk] Adds automatic settlement of orphaned chat runs in the cron cleanup job.

The PR appears safe to merge; no new actionable issue or outstanding previous finding was identified.

Summary

The PR adds a bounded sweep for orphaned chat runs, settles stopped runs without a controller, and records terminal statuses for Slack-search Assistant runs after outcome and response persistence.

The sweep guards against live locks, available replay, and concurrent controller claims.
Integration tests cover settlement, recovery races, and Slack-search terminal outcomes.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Unfinished chat run] --> B{Controller lock or replay available?}
  B -- Yes --> C[Leave for controller or recovery]
  B -- No --> D{Eligible after grace period?}
  D -- No --> C
  D -- Yes --> E[Recheck status and owner in transaction]
  E --> F[Settle run and clear matching chat marker]
  F --> G[Announce completion]

_{Reviews (4) · Last reviewed commit: "fix(knowledge): record the Slack run's s..."}

cubic-dev-ai

All reported issues were addressed across 7 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…sweeps where they stopped - The sweep takes each unowned leased run's chat lock under the run's own stream before settling it and releases it after the commit, so a reconnect can no longer lock the chat between the ownership check and the settle and then lose its claim; a reconnect that meets the fence retries. - A sweep examines at most 10k candidates and settles at most about 5k, resuming from a cursor saved in Redis and wrapping to the first run, so runs that cannot be settled yet never starve the ones after them. - Every settled run whose chat marker was released is announced, legacy runs included, so an open client stops showing the chat as busy.

waleedlatif1 · 2026-09-30T09:13:04Z

@greptile

waleedlatif1 · 2026-09-30T09:13:05Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T09:13:10Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

The Slack Assistant admits its own run row and only ever marked it as an error, so every completed or stopped Slack turn stayed active. It now records the terminal status once, after the turn ends, through the shared run-status update: complete on success, cancelled when its user stopped it (in Slack or in Sim), and error otherwise. Every other run-creating path already settles its run: interactive turns through their controller's finalize, and headless turns that admit their own run in the lifecycle's own finally.

cubic-dev-ai

All reported issues were addressed across 7 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T09:18:39Z

@greptile

waleedlatif1 · 2026-09-30T09:18:40Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T09:18:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 9 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

- The Slack Assistant now writes its run's one terminal status after its outcome and response are persisted, from the final outcome, so a failed save ends the run as an error instead of complete. - A Stop lookup that fails no longer skips that write: the turn is treated as not stopped, logged, and settled as an error. - The orphaned-run suite deletes the sweep cursor before each test and in teardown, so no later suite starts from its leftover position.

waleedlatif1 · 2026-09-30T09:43:47Z

@greptile

waleedlatif1 · 2026-09-30T09:43:49Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T09:43:55Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 9 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 added 4 commits September 30, 2026 01:18

vercel Bot temporarily deployed to Preview September 30, 2026 08:54 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.ts Outdated

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.ts

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/application/controls.ts

vercel Bot temporarily deployed to Preview September 30, 2026 09:12 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts

vercel Bot temporarily deployed to Preview September 30, 2026 09:18 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/application/slack-search/assistant.ts Outdated

Comment thread apps/sim/lib/knowledge/application/slack-search/assistant.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 30, 2026 09:43 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 merged commit cebe8a3 into staging Sep 30, 2026
32 checks passed

waleedlatif1 mentioned this pull request Sep 30, 2026

fix(mothership): explain withheld code and workflow results and bound workflow block logs #8459

Merged

waleedlatif1 deleted the fix/orphaned-run-settlement branch September 30, 2026 18:28

Conversation

waleedlatif1 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 30, 2026 •

edited

Loading

vercel Bot commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading