Repository navigation
acq-kits: convert the remaining six verify scripts to the shared verdict contract #463
Description
Tracking issue for the remaining work started in #460, so the staged conversion does not get lost once that PR merges.
Background
All eight kit scripts/verify ended with the same two-state verdict:
[ "$fail" -eq 0 ]That answers "did anything fail?" and cannot answer "did we actually check anything?". Two consequences, both measured:
- A run where nothing executed reported success. With
pass=0andfail=0the test is true, so the script printed "All checks passed." and exited 0 having verified nothing. - A skipped check was indistinguishable from a passing one. 35
skip/NOTE sites and 9warnsites across the eight scripts incremented no counter.openchamberandpaseodo defineskip()/warn(), but they are cosmeticprintf— sopaseocan print 21 yellow SKIPs and still claim its offline gate passed.
#460 adds integrations/isolation/acq-kits/verify-report.sh — five states (ok / bad / unver / skip / warn) with a verdict that is a function of a coverage record, exit 3 for "could not check" distinct from 1 for "failed" — plus scripts/tests/test_verify_contract.py enforcing it statically, since CI cannot run these scripts (they need a live sandbox).
It converts the two smallest kits only, deliberately, so the contract is reviewable before a 2,000-line mechanical edit.
Remaining work
Current state on main + #460, measured (not estimated):
| Kit | Lines | Gates | Legacy verdicts | skip/NOTE | warn | Status |
|---|---|---|---|---|---|---|
git-ssh-sign |
111 | — | 0 | 0 | 0 | converted (#460) |
usai-provider |
132 | — | 0 | 0 | 0 | converted (#460) |
agentic-coding-playbook |
167 | 1 | 2 | 1 | 0 | to do |
zscaler-ca-certificate |
167 | 1 | 2 | 1 | 0 | to do |
oci-engine |
303 | 2 | 2 | 0 | 0 | to do |
pi-coding-agent |
337 | 3 | 6 | 4 | 0 | to do |
openchamber |
641 | 3 | 6 | 8 | 2 | to do |
paseo |
864 | 3 | 6 | 21 | 7 | to do |
"Gates" counts info "Summary" blocks — the three-gate kits have an offline gate plus per-backend gates, and each needs its own verify_verdict call. A conversion that fixed only the first would leave the others reporting nothing.
UNCONVERTED in test_verify_contract.py lists exactly these six. It must reach empty, and test_every_kit_is_classified fails if a kit is in neither set — so a new kit cannot silently default to "converted" (that defect already happened once: oci-engine landed mid-review and the original subtraction-based logic classified it as converted).
Per-kit conversion checklist
- Replace the local
ok/bad/skip/warn/infodefinitions with. "$KIT_DIR/../verify-report.sh"+verify_reset. - Replace every
info "Summary"+printf+[ "$fail" -eq 0 ]block withverify_verdict "<gate name>". - Convert each silent skip to
unver(could not check — missing tool, absent credential, no network) orskip(not applicable in this configuration). This is the judgement call, and it is the whole point:unverdegrades the verdict,skipdoes not. Collapsing them is how a verify script claims coverage it does not have. - Move the kit from
UNCONVERTEDtoCONVERTEDinscripts/tests/test_verify_contract.py. - Run
pytest scripts/tests/test_verify_contract.py— it will reject a shadowed reporter, a surviving legacy verdict, a hand-rolledinfo "Summary"header, or a remaining bareecho "SKIP".
Step 3 is why I would not recommend doing all six in one PR: paseo alone has 21 sites, each needing a decision about whether that check could not run or does not apply. Mechanical for the structure, not for the semantics.
Suggested sequencing
- The two small ones (
agentic-coding-playbook,zscaler-ca-certificate) plusoci-engine— one PR, ~640 lines, low judgement load. pi-coding-agent+openchamber— one PR, three gates each.paseo— its own PR. 864 lines, 21 skip sites, 7 warns.
Related
- fix(acq-kits): give verify scripts a "could not check" verdict state #460 — the contract and the first two conversions.
- prime-agent kit: scripts/verify (pins + live assertions) #374 — prime-agent is the one kit with no verify script. When it gets one it should source the shared contract from the start rather than be converted later; worth a note on that issue.
AI-assisted (OpenCode), filed on operator instruction so the staged work is tracked rather than remembered. Advisory; sequencing is a maintainer call.
Activity
wz-gsa commentedon Oct 2, 2026
Related gap, noted here rather than filed separately since it is the same body of code.
No shellcheck hook exists in this repo, and the kit verify scripts are not linted by anything:
No shell linting among them, and
grep -rn 'scripts/verify'finds no reference inMakefile,.github/workflows/, or.pre-commit-config.yaml. So ~2,900 lines of shell across the eightscripts/verifyfiles are linted by nothing and run by nothing automated — they need a live sandbox, which CI cannot create.Measured baseline on two of them (info-level
SC2015/SC2016, no errors):Both counts are identical before and after the #460 conversion — the conversion added none.
I deliberately kept this out of #460: adding a linter to previously-unlinted code surfaces unrelated findings across every file and would have buried a 3-file change in noise. But "mentioned in a PR body" is not tracking, hence this note.
Worth considering alongside the conversion work rather than after it, since each kit is being touched anyway — a per-kit conversion PR could clear that kit's shellcheck findings in the same pass, and the hook could be added once the last kit is clean. That avoids a separate sweep over all eight files and avoids adding a hook that fails on day one.
Not urgent, and not a correctness bug:
SC2015(A && B || Cis not if-then-else) is the dominant finding, and in these scripts the pattern is used whereCrunning on a trueAis harmless. But it is the construct that hides theok/badasymmetry these conversions are cleaning up, so there is some overlap in intent.AI-assisted (OpenCode), on operator instruction.