Skip to content

ci(gate): split the CLI suite and the evaluate stage into parts that fit the sandbox cap - #466

Closed
debuggingfuture wants to merge 3 commits into
mainfrom
ci/split-long-gate-parts
Closed

debuggingfuture wants to merge 3 commits into
mainfrom
ci/split-long-gate-parts

Conversation

@debuggingfuture

@debuggingfuture debuggingfuture commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Every CLI-suite part and the evaluate stage now fit the 1800 s sandbox cap. Gate parts go from 29 to 36.

Problem & Insight

  • workspace.cli, features.binary-all and features.binary-data-plane each do one cold data-plane build (550–760 s) and then run about 440 ordinary CLI tests (1046–1126 s of wall clock). That adds up to 1620–1730 s on a good run, so any slower run hits ExecTimeout.
  • In evaluate, two entries account for most of the time, and both are compiles: the CLI build (eval-through-enforcement, 600–695 s) and the contextful-agent build (audit-read-waits-on-entry, 486–557 s). The stage ran 1368–1555 s before setup.

Take

  • CLI groups (assurance.gate.cli-groups): when a build links the data plane, the binary's integration modules are split into named groups in CLI_GROUPS (tools/ci/src/main.rs):

    • eval: eval, audit, derive, erase
    • pipe: pipeline, formal, author, holder, empty_replace

    Each group runs as workspace.cli-<group> and features.binary-<run>-<group>. The existing cli and binary-<run> parts skip the grouped modules and run every remaining test. Group parts use libtest substring filters and the rest part uses --skip, so a test with no group lands in the rest part; checked against all 460 current test names, no name matches two groups. The groups were balanced on measured per-module time, about 300 s each. The none and contextful-edge runs are not split: they pass in about 640 s.

  • Evaluate parts (assurance.gate.evaluate-parts): evaluate.binary runs the gate-tier entries whose test lives in the binary. evaluate.packages runs every other entry and probe. Each part reports its own floor verdict.

  • The spec, AGENTS.md, the part-count test and .github/rulesets/contextful-gate.proposed.json all list the 36 parts.

Expected budget

Part Compile Tests Projected
*.cli* / binary-<dp-run>* (3 each) ≤ 760 s ~375 s ≤ ~1200 s
evaluate.binary ~650 s ~40 s ~750 s
evaluate.packages — — ~850 s

Todos

  • Record the cold wall-clock time of each new part from this PR's gate run (target: ≥ 20% headroom, ≤ 1440 s).
  • Confirm three consecutive PRs with no ExecTimeout.
  • budget.full is one LTO release build (1145–1518 s compile, one timeout at 1820 s). Splitting it into parts does not apply; it needs its own fix (build parallelism or caching).

Closes #465

debuggingfuture and others added 2 commits October 11, 2026 22:49
…fit the sandbox cap

Under a build linking the data plane, the binary's eval and pipeline module
groups run as workspace.cli-<group> and features.binary-<run>-<group>; the
existing cli and binary-<run> parts run the remaining ordinary tests. The
evaluate stage dispatches evaluate.binary and evaluate.packages. The gate
lists 36 static parts.

Closes #465

Co-Authored-By: Claude <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://ci-split-long-gate-parts.spec.contextful.work (commit 61cfead)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://a19cdd6a.spec.contextful.work 61cfead 2026-10-11T14:51:23.239Z Visit the dashboard ↗

@debuggingfuture

Copy link
Copy Markdown
Member Author

Superseded by #467 and fractalboxdev/flare-dispatch#198. The dispatcher runs gate parts two at a time, so every extra part adds its own cold build to the gate's total time. A 45 min part deadline covers the slowest part (~1750 s) without adding parts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate parts features.binary-* and evaluate exceed the 1800 s sandbox cap

1 participant