Repository navigation
ci(gate): split the CLI suite and the evaluate stage into parts that fit the sandbox cap - #466
Closed
debuggingfuture wants to merge 3 commits into
Closed
debuggingfuture wants to merge 3 commits into
debuggingfuture wants to merge 3 commits into
Conversation
…fit the sandbox cap Under a build linking the data plane, the binary's eval and pipeline module groups run as workspace.cli-<group> and features.binary-<run>-<group>; the existing cli and binary-<run> parts run the remaining ordinary tests. The evaluate stage dispatches evaluate.binary and evaluate.packages. The gate lists 36 static parts. Closes #465 Co-Authored-By: Claude <noreply@anthropic.com>
🚀 Deploying Preview to Cloudflare 🚀Preview URL: https://ci-split-long-gate-parts.spec.contextful.work (commit 61cfead)This URL reflects your latest Preview deploymentPreview Deployments by commit
|
…patcher's 32-character name limit
Member
Author
|
Superseded by #467 and fractalboxdev/flare-dispatch#198. The dispatcher runs gate parts two at a time, so every extra part adds its own cold build to the gate's total time. A 45 min part deadline covers the slowest part (~1750 s) without adding parts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every CLI-suite part and the evaluate stage now fit the 1800 s sandbox cap. Gate parts go from 29 to 36.
Problem & Insight
workspace.cli,features.binary-allandfeatures.binary-data-planeeach do one cold data-plane build (550–760 s) and then run about 440 ordinary CLI tests (1046–1126 s of wall clock). That adds up to 1620–1730 s on a good run, so any slower run hitsExecTimeout.evaluate, two entries account for most of the time, and both are compiles: the CLI build (eval-through-enforcement, 600–695 s) and thecontextful-agentbuild (audit-read-waits-on-entry, 486–557 s). The stage ran 1368–1555 s before setup.Take
CLI groups (
assurance.gate.cli-groups): when a build links the data plane, the binary's integration modules are split into named groups inCLI_GROUPS(tools/ci/src/main.rs):eval: eval, audit, derive, erasepipe: pipeline, formal, author, holder, empty_replaceEach group runs as
workspace.cli-<group>andfeatures.binary-<run>-<group>. The existingcliandbinary-<run>parts skip the grouped modules and run every remaining test. Group parts use libtest substring filters and the rest part uses--skip, so a test with no group lands in the rest part; checked against all 460 current test names, no name matches two groups. The groups were balanced on measured per-module time, about 300 s each. Thenoneandcontextful-edgeruns are not split: they pass in about 640 s.Evaluate parts (
assurance.gate.evaluate-parts):evaluate.binaryruns the gate-tier entries whose test lives in the binary.evaluate.packagesruns every other entry and probe. Each part reports its own floor verdict.The spec,
AGENTS.md, the part-count test and.github/rulesets/contextful-gate.proposed.jsonall list the 36 parts.Expected budget
*.cli*/binary-<dp-run>*(3 each)evaluate.binaryevaluate.packagesTodos
ExecTimeout.budget.fullis one LTO release build (1145–1518 s compile, one timeout at 1820 s). Splitting it into parts does not apply; it needs its own fix (build parallelism or caching).Closes #465