[Michael McQuade (giraffesyo)]

Summary

The workflow hopperbench scenario was unstable: on an idle database at -jobs 100000, most runs drained in about 3 s and some took 20–48 s. Two problems in the finalize SQL (shared by hopperpgx and hoppersql) caused it.

1. The finalize statement could scan every running row for each result. Its retry and finished CTEs check j.state = 'running', which matches the predicate of the partial index hopper_jobs_running. The leader (or autovacuum) analyzes the backlog right after it is inserted, while nothing is running. With those statistics, scanning all of hopper_jobs_running looks like a single page, so the planner reads the whole index and loops over the result list for every row. A slow finalize leaves more completed jobs sitting in running, so the next statement has even more rows to compare. This repeats until the next analyze, 30 s or more later.

Fix: the running check compares the state as text (j.state::text = 'running'). No partial index predicate matches that expression, so results always find their rows by primary key. The two progress statements and the cancel-running statement find one running job by ID, and use the same check through isRunning(column). The cast is the standard form here: Postgres uses a partial index only for a condition on the identical expression, and checking the state after a primary-key match, = ANY ('{running}') or a subquery all still matched the index in EXPLAIN.

2. The finalize tail and dependent promotion read whole tables. The tail joined hopper_job_deps, hopper_jobs and hopper_batches to its own CTEs. The planner guessed hundreds of rows for those CTEs (the recursive cascade alone was estimated at 180 rows, and it is usually empty). hopper_job_deps also had no statistics, because nothing analyzed it during a short run. Under those estimates a sequential scan looked cheaper than index probes. The scan ran on every finalize and grew with the number of workflows in flight. The leader's promotion sweep had the same problem in a worse form: its LIMIT $2 makes a generic plan assume a tenth of the edges, so it scanned hopper_jobs once per chunk.

Fix:

• Every lookup keyed by the statement's own rows (cascaded, promoted candidates, edges, cleared progress, and the promotion candidates in workflowPromoteSQL) is now written as = ANY (ARRAY(SELECT ...)). The planner estimates ten keys for that form, so it always probes the index. ready looks each counted batch up by primary key in a LATERAL ... OFFSET 0 subquery, because an ANY there rescanned the counts once per batch.
• JobsMaintain now keeps hopper_job_deps (vacuum at 100k dead rows; server statistics target). Churn re-analyzes it at most every ten minutes, and at once if it has never been analyzed, because what the planner reads from it changes only with the shape of the workflows and each analyze makes every connection replan the finalize statement. The recursive step of the cascade can't use the ANY form, so it relies on these statistics. JobsMaintain reads all three tables' counters in one statement, by index.

No schema change, so there is no migration. PLAN §8 (plan stability, live-table maintenance) is updated. Now that workflow is stable, it is added to the perf gate, at 100,000 jobs (WORKFLOW_JOBS) so the gate's runtime stays reasonable.

Plans (EXPLAIN ANALYZE, generic; custom is the same shape)

Captured from a slow run of canary's workflow scenario, with 42,400 running rows and a batch of 100 results:

CTE finished
  ->  Delete on hopper_jobs j_1  (actual time=305.880..568.142 rows=100)
        ->  Nested Loop  (cost=0.28..8.59 rows=1) (actual rows=100)
              Rows Removed by Join Filter: 4239900
              ->  Index Scan using hopper_jobs_running on hopper_jobs j_1  (rows=1) (actual rows=42400 loops=1)
              ->  CTE Scan on r r_2  (actual rows=100 loops=42400)
CTE promoted
  ->  Hash Right Anti Join  (rows=19169) (actual rows=12)
        ->  Hash Join  (actual rows=119900)
              ->  Seq Scan on hopper_job_deps d_2  (actual rows=160000)
              ->  Seq Scan on hopper_jobs x  (actual rows=94900)
CTE edges
  ->  Hash Semi Join -> Seq Scan on hopper_job_deps d_4  (actual rows=160000)
Execution Time: 784.642 ms

Fixed, on the same statistics in a rebuilt state (40,000 running rows, 5,000 workflows in flight):

CTE finished
  ->  Nested Loop  (actual rows=100)
        ->  CTE Scan on r r_2  (actual rows=100)
        ->  Index Scan using hopper_jobs_pkey on hopper_jobs j_1  (actual rows=1 loops=100)
CTE promoted
  ->  Nested Loop Anti Join  (actual rows=12)
        ->  Index Scan using hopper_jobs_pkey on hopper_jobs j_3   (Index Cond: id = ANY (InitPlan))
        ->  Index Scan using hopper_job_deps_pkey / hopper_jobs_pkey
CTE edges
  ->  Index Scan using hopper_job_deps_pkey  (Index Cond: job_id = ANY (InitPlan))
Execution Time: 1.885 ms   (canary on that state: 643 ms)

The leader's promotion sweep over a 1,000-edge chunk went from 543 ms to 4.3 ms (generic).

Testing

• TestFinalizePlansIgnoreStaleStatistics (hopperpgx) builds 5,000 ten-step workflows, lets JobsMaintain analyze them before anything runs, and then claims 2,000 jobs. It explains seven statements under force_generic_plan and force_custom_plan: finalize with and without batches, follow-up and sweep promotion, both progress statements, and cancel-running. It asserts that none of them uses hopper_jobs_running and none sequentially scans hopper_jobs, hopper_job_deps or hopper_batches. On canary's SQL it fails with 3 hopper_jobs_running plans and whole-table scans in both modes.

• WorkflowsConcurrent (drivertest, both drivers) is a chaos test. It runs 100 fan-out/fan-in workflows (one workflow in five has a failing middle step) through 8 workers with small claims, so middle steps split across statements. Each worker follows up its own flushes, the leader's sweeps run alongside, and every statement is sent a second time. It asserts:

  • every result applies exactly once;
  • every job has one history row;
  • no claimed job still has a live dependency;
  • each failure cancels the step after it;
  • batch failure counts are right;
  • each workflow fires one completion callback;
  • no edges are left behind.

  Spot-checked against mutations. It fails if either promotion statement loses its live-dependency check, if the cascade misses pending jobs, or if edges of cascaded jobs are kept. It passed 15/15 runs under -race on both drivers.

• DepsTableMaintained (drivertest, both drivers): JobsMaintain analyzes hopper_job_deps after a burst of edges. Fails on canary.

• Existing finalize, fence, batch and fan-in concurrency tests pass. make check passes.

hopperbench

Base 8822336 against head, alternating (the side that goes first switches each round), 5 rounds, -jobs 100000, latency with 1,000 samples. Each run started only when the 1-minute load average was under 8. The table shows the mean of the faster half of each side's runs (hopperbench -compare):

scenario	base	head	change
throughput	185,025 jobs/s	182,913 jobs/s	−1.1%
mixed	188,564	190,244	+0.9%
scheduled	166,593	175,263	+5.2%
retry	86,619	86,837	+0.3%
tagged	141,907	144,159	+1.6%
latency	p50 0.33 ms, p99 0.85 ms	p50 0.34 ms, p99 0.77 ms	ok
loaded	39,742	39,802	+0.1%
insert	388,568	389,551	+0.3%
copy	648,975	597,014	−8.0% (path unchanged; 0.16 vs 0.18 s runs)
workflow	32,724	100,435	+207%

workflow drain time, median and spread:

	runs	median	range
base, this comparison	5	3.20 s	2.88–3.63 s
base, all local runs	13	3.24 s	2.88–33.1 s (2 runs at 23 and 33 s)
head, this comparison	5	1.04 s	0.95–1.22 s
head, all local runs	13	1.03 s	0.95–1.28 s

Checklist

• make check passes
• No cryptography was added (tests run with GODEBUG=fips140=only)
• New or changed SQL in the claim, finalize, rescue or leader paths has a concurrency or chaos test
• Behavior changes are reflected in docs/PLAN.md

Bench after review round 2 (workflow and throughput, alternating with 8822336, 5 rounds, load under 8)

	base	head
workflow drain	3.13–29.7 s (median 3.30)	0.91–1.02 s (median 0.95)
workflow, faster half	31,297 jobs/s	107,828 jobs/s (+245%)
throughput, faster half	185,122 jobs/s	188,697 jobs/s (+1.9%)