Skip to content

perf(daemon): make scheduling passes linear in the number of tasks - #377

Open
arcanis wants to merge 1 commit into
mael/tasks-persistent-devfrom
mael/daemon-scheduling-perf
Open

arcanis wants to merge 1 commit into
mael/tasks-persistent-devfrom
mael/daemon-scheduling-perf

Conversation

@arcanis

@arcanis arcanis commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Issue: a trivial task across 1000 workspaces took 16-20s. After every command, including each output line, the coordinator rescanned every task and allocated keys for each prerequisite check.

Fix:

  • Passes iterate prepared tasks once, with precomputed satisfied/failed sets and borrowed-key lookups.
  • The failure scan is skipped when nothing failed.
  • Commands that can't change task states don't trigger a pass.
  • Standalone daemons don't register taskfile watchers.

1000 tasks now take ~1-2s. Stacked on #376.


Note

Medium Risk
Scheduling and prerequisite propagation logic changed on the critical coordinator path; incorrect skips or set membership could delay spawns or miss failure propagation, though behavior should be equivalent if the command classification is complete.

Overview
Improves daemon performance for large monorepos by cutting redundant scheduling and expensive startup work.

The coordinator skips drain_ready_tasks after read-only commands (notably TaskOutput and other subscription/query/file-watch paths) so high-volume output no longer triggers full graph scans.

find_ready_tasks / find_tasks_to_fail are rewritten to build satisfied/failed sets once per pass and check prerequisites with borrowed keys instead of allocating a ContextualTaskId per edge; failure propagation returns immediately when nothing failed.

Inline (standalone) daemons detect port_tx and disable taskfile hot-reload watching (TaskfileWatcher::set_enabled(false)), avoiding O(workspaces) FSEvents registration that was dominating startup.

Reviewed by Cursor Bugbot for commit e5b5e7f. Bugbot is set up for automated code reviews on this repo. Configure here.

Running a trivial task across 1000 workspaces took 16-20s. The
coordinator spent most of that rescanning every resolved task (and
allocating keys for each prerequisite) after every command, including
each output line.

- Scheduling passes iterate prepared tasks once, compute the satisfied
  and failed sets once, and look prerequisites up by borrowed keys.
- The failure scan is skipped when nothing failed.
- Commands that can't change task states (output lines, queries,
  subscriptions) don't trigger a pass.
- Standalone daemons don't register taskfile watchers.

1000 tasks now take ~1-2s.
@arcanis
arcanis force-pushed the mael/tasks-persistent-dev branch from 55deb1c to 86a2038 Compare October 9, 2026 13:59
@arcanis
arcanis force-pushed the mael/daemon-scheduling-perf branch from 02b4450 to e5b5e7f Compare October 9, 2026 13:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant