feat(isolation): validate kit templates under acq-kits/examples/

[basiliskus]

Summary

Makes integrations/isolation/acq-kits/examples/ a container of copy-and-rename kit templates: validate-kits.py checks each examples/<name>/ like a kit but keeps it out of the kits.yaml registry.
ADR 0005 records why scope-layer kits ship as templates (hybrid/v1 has no parameters) and where they live (placement B).
No template ships here; the first one, team-kit, and the scope-layers pattern doc follow in a stacked PR.

Reviewer ask: is placement (B), acq-kits/examples/<name>/ exempt from the registry, the right home? The alternative (A) is acq-kits/<name>/ with a kits.yaml entry marked TEMPLATE and no validator change; ADR 0005 lists both. Switching to (A) means dropping this PR and moving the template up one level in the next.

Eligibility & Accountability

• I am a federal employee or contractor (PRs merge with @GSA-TTS/agentic-coding-team code-owner approval; eligibility is attested here, not authenticated). Anyone may open issues without this.
• I understand this change and can explain it in my own words — I am the author of record and accountable for it.
• By opening this PR, I attest I have the right to release this work under the repository's CC0-1.0 dedication.

AI Assistance Disclosure

• AI tools were used in this contribution.
• I verified all AI-generated content: no fabricated/hallucinated APIs, dependencies, citations, or placeholder content; any AI-suggested dependency was confirmed to exist.
• Any inherited/reused third-party material retains its original license and is identified (not represented as public domain).

Type of Change

• Infrastructure/tooling change
• Documentation update

Motivation

Every team adopting acq builds the same team scope layer. A template CI keeps schema-valid is more useful than prose alone, but a template is not a kit acq applies, so it must not sit in the registry of deployable kits. This PR adds the home and the gate; the content lands separately so the placement question can be answered on its own.

What lands:

• acq-kits/validate-kits.py: skips examples/ in the kit walk; validates each examples/<name>/ with the same per-kit checks (schema, files[].source resolution, env names, README) under an examples/ prefix; leaves templates out of the kits.yaml cross-check, so a template listed there fails ("registry lists unknown kit"). The walk skips only named tool droppings (__pycache__, .pytest_cache, .ruff_cache, .mypy_cache); any other directory, dot-named or _-prefixed, is validated, because acq can fetch any directory through a dir= ref. Two tree-level checks: no two kits or templates may declare the same name: (a copied directory that kept the original name is the likeliest copy-and-rename slip, and the registry cross-check could not see it), and a tree with no kits or templates fails instead of passing with nothing validated.
• scripts/tests/test_validate_kits_examples.py: 22 tests on generated fixture trees, including duplicate names (kit/kit, template/template, kit/template), empty or missing trees, and dot-named kits and templates; none depend on a real template.
• integrations/isolation/docs/decisions/0005-kit-templates.md: templates vs a parameterized kit vs doc only, and placement A vs B. Numbered 0005 because feat(isolation): add serviceGateways kit vocabulary #416 and docs(isolation): add ADR 0004 for neutral model-provider discovery #423 claim 0003 and 0004; happy to renumber.
• acq-kits/README.md: a Templates section, separate from the Available kits table.

Testing

• Ran make validate locally (all checks pass)
• Ran make test (if applicable): 429 passed; scripts/tests/test_validate_kits_*.py: 95 passed
• Tested with at least one real scenario: the validator on this repo's real tree with no template present (All 10 acq-kits valid.); the stacked PR exercises it with one (All 10 acq-kits + 1 template(s) valid.)
• Examples are concrete and realistic
• Commit messages follow Conventional Commits format

Also run: validate-kits.py --strict, ruff check scripts/, ruff format --check scripts/, make generate-check.

Safety Checklist

• No secrets, API keys, tokens, or passwords included
• No PII (names, emails, SSNs, addresses)
• No CUI (Controlled Unclassified Information)
• No internal URLs, hostnames, or system paths
• No customer data or proprietary information
• Pattern defines prohibited_content in output contract (if applicable): not applicable, no SKILL.md pattern

Quality Checklist

• Plain language used (Grade 10 or below preferred)
• Examples included
• Prerequisites clearly stated

Validation Results

make validate
✓ All validators passed

Related Issues

Relates to #421 (placement option B)
Relates to #416 and #423 (ADR numbering; validate-kits.py and acq-kits/README.md overlap with #416)

Reviewer Notes

Reviewers should verify:

• Placement (B) under acq-kits/examples/ is the preferred option
• The validator keeps templates out of the registry and still validates them
• make validate passes in CI

[wz-gsa]

Reviewed the validator change, ADR-0005, and the new test file. Two should-fix findings, one nit. Both findings are pre-existing rather than introduced here, but both live in the function this PR edits, and one gets more likely because of copy-and-rename templates.

Everything below was run against the PR's actual validate-kits.py, not read off the diff.

---

Should-fix 1 — two kits may declare the same name: and validation passes

acq-kits/a/spec.yaml   name: samename
acq-kits/b/spec.yaml   name: samename
kits.yaml              kits: { samename: … }

$ python3 integrations/isolation/acq-kits/validate-kits.py --root /tmp/duptest
  OK  a
  OK  b

All 2 acq-kits valid.
exit=0

Cause is the set collapse in the registry cross-check:

present = set(kit_names)     # kit_names is a LIST; duplicates vanish here
missing = present - listed
extra   = listed - present

Both differences come out empty, so two directories claiming one identity satisfy a single registry entry. Since acq fetches kits by explicit dir= URL (quickstart common.sh:42-46), a and b are genuinely distinct fetchable kits reporting the same name, and nothing flags it.

Why this belongs on this PR specifically. ADR-0005 makes templates copy-and-rename sources, and the README instruction is to rename "both the directory and name:". Renaming the directory and forgetting the name: is the single most likely authoring slip the template workflow introduces — and it is exactly the case that currently passes.

Suggested fix, verified:

seen: set[str] = set()
for name in kit_names:
    if name in seen:
        all_errors.append(f"kits.yaml: duplicate kit name {name!r} declared by more than one kit directory")
    seen.add(name)
present = seen

duplicate pair  -> "duplicate kit name 'samename' declared by more than one kit directory"
distinct pair   -> no errors (control)

---

Should-fix 2 — an empty kits tree reports success

if not kit_dirs and not template_dirs:
    print(f"No kits found under {kits_dir}")
    return 0

$ python3 validate-kits.py --root /tmp/emptykits2     # valid schema, empty acq-kits/
No kits found under …/integrations/isolation/acq-kits
exit=0

Zero kits validated, zero templates validated, exit 0. This PR extends the condition (correctly — a tree holding only templates must not short-circuit) without changing the verdict.

Honest severity: latent, not a live CI hole. I checked the realistic ways to hit it:

• CI runs make validate-kits with no --root against a populated tree.

• Running from a subdirectory fails earlier, on the schema check, so the obvious developer mistake is already caught:

  $ cd integrations && python3 ../integrations/isolation/acq-kits/validate-kits.py
  ERROR: schema not found: …/integrations/schemas/kit-hybrid-v1.schema.json
  exit=1
  

So reaching return 0 takes a deliberate --root, or moving/renaming acq-kits/ while leaving schemas/ in place. Worth fixing because the gate runs in both CI and pre-commit, and because of the asymmetry it creates: a missing kits.yaml is a hard FAIL (kits.yaml registry not found), while a missing kits tree is a pass. The weaker condition gets the stronger verdict.

The file only has exit codes 0 and 1, so the minimal honest change is return 1 with a message saying nothing was validated — rather than inventing a third code here.

---

Nit — the new test file has no empty-tree case

test_validate_kits_examples.py is thorough: 13 assertions across both exit 0 and exit 1, including a mixed valid/invalid tree (:130-133) and _wip- prefixed directories (:120-123). That is better coverage than I expected.

The one degraded path it does not assert is the empty tree — which is why the return 0 above survives a change to its own guard. If Should-fix 2 is taken, a test asserting "no kits and no templates is not a pass" would pin it.

---

Two concerns I checked and withdrew

Recording these because both look plausible from the diff and neither is real:

• _is_tool_dropping skipping any dotted directory as a hiding place. I planted an invalid kit at acq-kits/.sneaky/ expecting silence. It was validated and failed: exit=1, .sneaky: schema: 'mode' is a required property (at files/0). No bypass.

• Smuggling a real kit under examples/ to dodge the registry cross-check. Traced how acq resolves kits: explicit dir= URL constants (common.sh:42-46), and kits.yaml has zero references anywhere in acq or its backends. acq never scans the tree, so nothing under examples/ can be applied. The exemption is correct.

That second check also confirms ADR-0005's strongest claim — "A template that is listed in kits.yaml fails validation … so the separation is enforced, not advisory" — is accurate as written, and the PR tests it (:81-82).

---

What I verified as sound

• Templates get the same validate_kit as kits (schema, files[].source resolution, env names, README presence), reported under an examples/ prefix. Not a weaker tier.
• The kit_dirs walk correctly excludes TEMPLATES_DIR and tool droppings, and template_dirs is discovered separately so a templates-only tree is still validated rather than short-circuited — the comment at that site says exactly that, and it is true.
• ADR-0005's "live, harmless value" rationale (commented-out YAML is never schema-checked, so a template made of comments rots silently) is a real argument and matches the placement decision.
• This PR adds no examples/ templates, only validator support for them. I went looking for whether the escape-primitive gaps I raised on GSA-TTS/agentic-coding-patterns#416 would propagate through a template compose file into every derived kit — they would, but no template ships one here, so it is out of scope for this PR. Worth keeping in mind when #441/#442 land actual templates.

---

AI-assisted review (OpenCode), prepared on operator instruction. Advisory. Neither finding blocks this PR in my view — Should-fix 1 is the one I would take, since the template workflow is what makes it likely.

[basiliskus]

Thanks, both reproduced against this branch (duplicate name: exits 0; empty and missing acq-kits/ exit 0). Taking both: the duplicate check now spans kits and templates together, since copy-and-rename between templates is the same slip, and an empty tree fails with a message that nothing was validated. Tests added for both. Agree the #416 compose-file concern doesn't apply yet; neither #441 nor #442 ships a compose file.

[wz-gsa]

AI-assisted re-review (OpenCode), advisory — needs a human to confirm before it drives a change. Reviewed head ee1be35.

Both findings are fixed. I re-ran my original reproductions against this head rather than reading the diff.

Should-fix 1 — duplicate name: — fixed, and the extension is correct

The set collapse is gone; name_owners now maps name → every declaring directory, checked at validate-kits.py:411-413.

Reproduced my original case:

$ python3 integrations/isolation/acq-kits/validate-kits.py --root /tmp/duptest3
FAIL:
  - duplicate kit name 'samename' declared by: a, b
exit=1

Control (distinct names) produces no duplicate error.

Your extension to span kits and templates works and matters more than the original finding — the cross-container case is the actual copy-and-rename slip:

  - duplicate kit name 'samename' declared by: a, examples/tpl

The error names the owning directories rather than just the colliding name, which is what makes it actionable.

Should-fix 2 — empty tree — fixed

validate-kits.py:362-366 returns 1. Both cases I reported:

# empty acq-kits/
ERROR: no kits or templates found under .../acq-kits; nothing was validated
exit=1

# acq-kits/ absent entirely
ERROR: no kits or templates found under .../acq-kits; nothing was validated
exit=1

Templates alone still validate (test_templates_are_validated_when_no_kit_dirs_exist), so the gate doesn't over-trigger.

Tests

test_validate_kits_examples.py covers both findings across four cases — test_duplicate_name_across_kits_fails, test_duplicate_name_across_templates_fails, test_template_reusing_a_kit_name_fails, test_empty_tree_is_not_a_pass, plus test_missing_kits_dir_is_not_a_pass.

PYTHONPATH=scripts pytest scripts/tests/test_validate_kits_examples.py -q  -> 22 passed
PYTHONPATH=scripts pytest scripts/tests/                             -q  -> 429 passed
python3 scripts/validate_repo.py                                         -> all validators passed

No further findings. Agreed on the #416 compose-file point — not applicable until a template ships one.

---

Method: re-ran the original reproductions (duplicate pair, distinct-name control, cross-container duplicate, empty tree, absent tree) against ee1be35's validate-kits.py, plus the test suite and repo validators. No sandbox created.

[basiliskus]

@wz-gsa just to confirm there is nothing to act on my side from the last review. I'm not sure if your review skill is missing a verdict?

[wz-gsa]

Checked whether this needed a re-review and it does not — the force-push is a rebase, not a content change.

git diff ee1be35 b1cb1972 -- integrations/isolation/acq-kits/validate-kits.py \
                             scripts/tests/test_validate_kits_examples.py
(empty)

Your own files are byte-identical between the old and new head; the whole delta is #436's merged files arriving underneath. My re-review of ee1be35 therefore still stands as the current verdict: both findings fixed, 22 new tests pass, 429 full suite, validators clean, no new findings.

Not re-running the checks on an unchanged tree — this is waiting on a maintainer, not on me.

[wz-gsa]

Approving. Converting my 2026-10-07 re-review into a formal review — it was posted as a comment, which left this at REVIEW_REQUIRED and reading as unreviewed. My fault, not yours.

Still applies at the current head. b1cb1972 → 57b3d128 is a rebase; this PR's own files are byte-identical:

git diff b1cb1972 57b3d128 -- integrations/isolation/acq-kits/validate-kits.py \
                              scripts/tests/test_validate_kits_examples.py
(empty)

Both findings from my first review are fixed, verified by re-running the original reproductions rather than reading the diff:

Duplicate name: across kits. The set() collapse is gone; name_owners maps name → every declaring directory (validate-kits.py:411-413). My original case now fails, and the error names the owning directories rather than just the colliding name, which is what makes it actionable:

FAIL:
  - duplicate kit name 'samename' declared by: a, b
exit=1

Distinct names produce no error. Your extension to span kits and templates catches the likelier slip — a copy-and-rename between containers:

  - duplicate kit name 'samename' declared by: a, examples/tpl

Empty tree reported success. validate-kits.py:362-366 now returns 1 for both an empty acq-kits/ and an absent one, and templates-only still validates (test_templates_are_validated_when_no_kit_dirs_exist), so the gate does not over-trigger.

Tests and validators at the head I reviewed: test_validate_kits_examples.py 22 passed, full suite 429 passed, scripts/validate_repo.py all validators passed.

Agreed on the #416 compose-file point — not applicable until a template ships one.