Skip to content

feat(oci-engine): add rootless podman OCI-engine capability kit - #437

Merged
mogul merged 13 commits into
mainfrom
feat/oci-engine-kit
Oct 2, 2026
Merged

mogul merged 13 commits into
mainfrom
feat/oci-engine-kit

Conversation

@mogul

@mogul mogul commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

AI-assisted. Adds a new neutral acq mixin kit, oci-engine, that provides a rootless podman OCI container engine as an opt-in capability. This implements the OCI-engine-kit direction from ADR-0030 (in the sibling agentic-coding-quickstart repo): move the rootless-podman provisioning out of the msb backend adapter and into a neutral, backend-agnostic kit that every acq backend applies through the shared hybrid/v1 vocabulary, off by default.

It is consumed opt-in by acq: the quickstart side already added an ACQ_ENABLE_OCI_KIT selection gate that fetches and validates a kit at the pinned patterns ref requiring exactly schemaVersion: "hybrid/v1", kind: mixin, name: oci-engine. Until this kit merges and is pinned, that opt-in no-ops by design.

What it does

  • Install (create-time, root): installs podman + rootless prerequisites (podman-compose, fuse-overlayfs, uidmap, passt, slirp4netns) via apt/dnf/apk; selects a storage driver that works on an overlay root (overlay+fuse-overlayfs, else vfs); writes a Docker-Hub-first registry config (short-name-mode=enforcing by default); shadows the base image's dead-socket docker CLI with a docker→podman wrapper.
  • Startup (every boot): re-grants /dev/net/tun + /dev/fuse to the agent (root:agent, 0660) since /dev is re-created each boot; and runs a rootless layer-mount self-test that writes a user-level vfs storage fallback if the overlay+fuse-overlayfs combo is rejected under rootless.
  • Runs rootless as the agent, never root; daemonless (no dind, no socket). Preserves the ADR-0020 security posture (rootless, explicit minimal device handling, least-privilege enforcing short-name mode).

Security posture (ADR-0020)

Rootless engine; only install/config and the device grant need root. Exactly two device nodes exposed, group-scoped 0660 (not world-writable). Registry resolution is Docker-Hub-first with short-name-mode=enforcing (least-privilege / prompt-injection defense: ambiguous unqualified names fail closed instead of being silently substituted). No secrets. Egress: none declared — the OS package mirrors are already in the balanced baseline core tier (isolation ADR-0002); a strict tier fails the install soft.

Verification

  • make validate-kits (validate-kits.py --strict): All 8 acq-kits valid, no errors/warnings.
  • sh -n clean on all 4 /bin/sh scripts; bash -n clean on the verify script; ShellCheck (severity=warning) clean.
  • Fail-soft proven: with a stubbed failing package manager and no podman on PATH, the install script prints a clear warning and exits 0 (does not abort sbx create).
  • scripts/verify is a host-side live check (needs sbx; skipped in CI, like the zscaler kit's verify) — not run here (no sandbox-capable host).

Review

This kit was written and then put through an independent adversarial review before this PR. That review found and this branch fixed: (1) a blocker where set -eu + bare package-manager calls defeated the documented fail-soft and would have failed sbx create; (2) a dropped rootless vfs recovery step vs. the reference implementation; (3) a verify script that could pass green on a rootful engine. All three are fixed in commit 3d9010b, with a docs/decisions/ note recording the port and the fail-soft + vfs-fallback choices.

Rollback

Revert this PR. The kit is additive and off by default; nothing selects it until the quickstart side pins this ref and an operator sets ACQ_ENABLE_OCI_KIT. The msb adapter's existing podman provisioning remains the active OCI mechanism in the meantime.

Related

mogul and others added 2 commits September 21, 2026 21:59
Add the patterns-side `oci-engine` neutral acq mixin kit
(schemaVersion "hybrid/v1", kind mixin, name oci-engine) under
integrations/isolation/acq-kits/oci-engine/. It provides a rootless
container engine (podman) as a neutral, opt-in capability, implementing
the isolation ADR-0030 decision to move that provisioning out of the msb
backend adapter (sibling agentic-coding-quickstart repo) and into a
backend-agnostic kit. acq applies it only when the operator opts in via
ACQ_ENABLE_OCI_KIT on the quickstart side.

The kit ports the working msb-adapter logic into the neutral vocabulary:
an install-phase script (root, create-time once) installs podman + the
rootless prerequisites, selects a storage driver that works on an overlay
root (overlay+fuse-overlayfs, else vfs), writes a Docker-Hub-first
registry config, and shadows the dead-socket base docker CLI with a
docker->podman wrapper; a startup-phase script (root, every start)
re-grants /dev/net/tun + /dev/fuse to the agent because /dev is a devtmpfs
re-created each boot. Both scripts are idempotent, POSIX sh, and fail
soft.

Preserves the ADR-0020 security decisions: the engine runs rootless as
the agent (never root; only install/config needs root), no
docker-in-docker daemon lifecycle, explicit group-scoped (0660, not
world-writable) device handling, and Docker-Hub-first registry resolution
with short-name-mode="enforcing" as the DEFAULT (least-privilege /
prompt-injection defense) rather than permissive.

Declares no caps.network.allow: the OS package-mirror host depends on the
base distro and is already reachable under acq's balanced egress baseline
(ADR-0002), so hardcoding a distro mirror would be wrong and redundant;
a strict tier fails the install soft. Ships a host-side scripts/verify
live check (rootless podman build FROM scratch + docker->podman wrapper),
skipped in CI. Registers the kit in kits.yaml with a prose parity note.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
…ess verify

Address three findings from adversarial review of the oci-engine kit.

Fix 1 (blocker) — make the install phase FAIL SOFT. `set -eu` plus bare
apt-get/dnf/apk calls aborted the install script on the first non-zero
return, which fails `sbx create` (a dead sandbox) — the opposite of the
documented fail-soft promise. Run each distro install inside a subshell
under its own `set -e` and swallow the exit; a failure now falls through
to the single warn-and-exit-0 `command -v podman` re-check so provision
continues. Verified by stubbing failing package managers on PATH: the
script prints the warning and exits 0.

Fix 2 — restore the reference adapter's rootless recovery. Add a second
startup step run as the agent (uid 1000),
files/home/oci-engine-vfs-fallback.sh: a real rootless layer-mount
self-test (podman build FROM scratch) and, on failure, a user-level
~/.config/containers/storage.conf selecting vfs, then retry. Idempotent
and fails soft (always exits 0). Covers bases whose overlay+fuse-overlayfs
combo is rejected under rootless, where `podman info` passes but a layer
mount fails.

Fix 3 — make scripts/verify actually prove rootless. Run the agent-phase
checks with an explicit `-u agent` (matching the reference), and hard-fail
step 4 when podman does not report rootless=true (falling back to a
non-zero agent uid assertion when the format key is unavailable) instead
of printing PASS regardless.

Also add oci-engine/docs/decisions/ recording the ADR-0030 port and the
fail-soft + vfs-fallback behavioral choices, matching the other kits.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
mogul and others added 3 commits September 22, 2026 05:07
…dapter OCI off

Rework the kit's host-side live check so it demonstrates the kit — not the
msb adapter's legacy auto-install — actually provisions rootless podman.

- Switch from raw `sbx create --kit` to the real path: `acq --backend msb
  create shell <ws> --kit <this kit>` + `acq exec` (which runs as the agent
  user), exercising kit-translate + kit apply end-to-end. msb-only, since msb
  is acq's default backend and the backend that demonstrates OCI behavior
  (ADR-0020); sbx base-image selection is out of scope.
- Export ACQ_MSB_ENSURE_OCI=0 for the run so acq's still-active adapter
  auto-install cannot mask a broken kit with a false pass. With it off, podman
  and especially the /usr/local/bin/docker wrapper are attributable to the
  kit. When the adapter auto-install is removed (ADR-0030), the export becomes
  a no-op and this stays the standing regression test.
- Derive the sandbox name the way acq does (source acq under ACQ_SOURCE_ONLY,
  call derive_name) for correct exec/cleanup, with a slug fallback.
- Keep the rootless assertions (rootless=true, else non-zero agent uid) and the
  FROM-scratch layer-mount build. Update README's Verifying section to match.

Companion quickstart work (add ACQ_ENABLE_OCI_KIT gate, retire the adapter's
OCI auto-install, repoint verify-backends) is tracked in the quickstart repo.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
…an info stderr

msb's exec transport returns guest stdout as CRLF, so `in_sbx` captured a
trailing \r that (a) broke exact-string compares — e.g.
"/usr/local/bin/docker\r" != "/usr/local/bin/docker", a false FAIL in the
docker-wrapper check — and (b) when printed, moved the cursor to column 0 so
the next char overwrote the line (the garbled ") PASS" / ") FAIL" output).
Strip \r once in in_sbx (the same fix the quickstart's run_acq uses).

Also surface podman's actual stderr when the step-4 `podman info` check fails,
so a failure is diagnosable instead of a black box.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
…ves the kit

Two changes so verify actually proves the KIT provisions podman, and fails
legibly when it does not:

- Pin a neutral base image that ships neither docker nor podman
  (VERIFY_OCI_IMAGE, default docker/sandbox-templates:shell) via acq's
  --image. The msb default base (…:shell-docker) already bundles a container
  engine and puts the agent in the docker group, so podman/docker present
  there is NOT attributable to the kit — defeating ACQ_MSB_ENSURE_OCI=0. On a
  neutral base the kit is the only thing that can produce a working rootless
  podman (it installs from the OS mirror at create). Override with
  VERIFY_OCI_IMAGE=<ref>.
- Make "podman present" a HARD GATE (step 3): if the kit's fail-soft install
  did not land podman, dump the oci-engine lines from the create log (the real
  reason — unreachable mirror, unsupported base) and stop, instead of
  cascading into confusing rootless FAILs. This is what the last run needed:
  on …:shell-docker the base had no engine the kit installed, the fail-soft
  install exited 0, and podman was silently absent.

README Verifying section updated to document the neutral-base requirement and
the VERIFY_OCI_IMAGE override.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul
mogul marked this pull request as draft September 22, 2026 19:48
…om install

Live testing on a base that already bundles a container engine
(docker/sandbox-templates:*-docker runs a real dockerd) surfaced two issues
in the install script; fix both.

- Functional-engine early-out: before doing anything, if `docker info`
  succeeds the base already has a WORKING engine, so leave it alone (do not
  install podman or shadow the working docker) and exit 0. Probe `docker info`
  (function), NOT merely a docker BINARY: many bases ship a docker CLI whose
  daemon socket is dead in the sandbox (the exact case this kit fixes), which
  must NOT early-out and correctly proceeds to install podman.
- Decouple config/wrapper from the package step: the storage/registry config
  and the docker->podman wrapper now run whenever podman is PRESENT at that
  point (installed by the kit OR pre-existing), not only when the kit ran the
  package install. Previously a skipped/failed package step also skipped the
  config, which is why /usr/local/bin/docker was absent on a base without
  podman.

Both paths keep the fail-soft posture (exit 0). Verified via stubbed shells:
no-engine+failing-mirror -> fail-soft; functional docker -> early-out;
dead-socket docker -> proceeds to install. spec.yaml, README, and the kit's
docs/decisions updated to describe the early-out and the decoupling.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@wz-gsa

wz-gsa commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

AI-assisted adversarial review (nexus-agents pr_review, 5-role panel: architect/security/devex/catfish/scope_steward), advisory — needs a human to confirm before it drives a change. Ran in two rounds; the second round re-verified every finding from the first directly against the diff. Result: 5-0 request_changes, all voters completed (no timeouts), high confidence (0.82–0.86).

Architecture is sound (Scope Steward did a build-vs-buy pass: right layer per ADR-0030, podman/fuse-overlayfs/passt/slirp4netns are the correct bought components, not a reimplementation, and this consolidates provisioning that's currently duplicated in the msb adapter — net DRY-positive). The blocking issues are all in the implementation's verifiability and a few concrete correctness/security defects, confirmed independently by multiple voters with line citations:

1. Fail-silent, not fail-soft (all 5 voters, highest severity). Every failure path in oci-engine-install.sh — unsafe OCI_ENGINE_PKGS (line 20), no package manager found (line 46), apt-get/dnf/apk install failure (lines 33/38/43), podman still absent after install (line 52) — exits 0 with only a stderr warning. oci-engine-vfs-fallback.sh does the same on every one of its failure branches. No marker file is written, and the kit ships no scripts/verify, unlike this repo's established per-kit convention. Net effect: acq reports a sandbox with the kit successfully "applied" while docker run may be completely absent, and there's no way for CI or an operator to tell.

2. Real supply-chain gap: package-manager flag injection (Security, DevEx, Architect). OCI_ENGINE_PKGS's sanitizer (case "$OCI_ENGINE_PKGS" in *[!A-Za-z0-9._+\ -]*)) permits a leading - in any token, so OCI_ENGINE_PKGS="podman --allow-unauthenticated" passes validation and word-splits straight into apt-get install -y --no-install-recommends $OCI_ENGINE_PKGS — disabling package signature verification. The dnf/apk equivalents (--nogpgcheck, --allow-untrusted) pass the same filter. A sanitizer whose stated job is rejecting unsafe input, with a known bypass, is worse than no sanitizer.

3. Unconditional elevated device grant (Security, DevEx, catfish). oci-engine-grant-devs.sh chowns/chmods /dev/net/tun and /dev/fuse to the agent group on every boot with no command -v podman check — contrast oci-engine-vfs-fallback.sh, which does guard on podman. So a fully failed install (silently, per #1) still widens device access every boot for a capability that doesn't exist. Security flagged /dev/net/tun specifically as egress-relevant (enables in-guest tunneling around guest-side filtering), not just cosmetic.

4. Storage driver permanently mis-pinned from a boot-transient probe (Architect, DevEx, catfish). oci-engine-install.sh:64 selects overlay+fuse-overlayfs based on [ -e /dev/fuse ] at create time, then guards against ever rewriting it (grep -q '^[[:space:]]*driver'). But the kit's own oci-engine-grant-devs.sh exists specifically because /dev is recreated every boot — so a single transient miss at create time permanently bakes in the slow vfs driver with no recovery path, ever.

5. Missing caps.network egress declaration (DevEx, catfish). spec.yaml declares files/commands/environment and no caps block, despite the install needing distro package mirrors and the kit's whole purpose needing docker.io/registry-1.docker.io/auth.docker.io. On any backend enforcing an egress allow-list, install fails — and per #1, does so invisibly.

6. Unbounded podman build on the boot path, no timeout (catfish). The rootless self-test in oci-engine-vfs-fallback.sh runs a real image build on every boot with no timeout — a wedged storage lock or newuidmap stalls the entire boot sequence.

Happy to see this land once the observability gap (marker file + a scripts/verify matching the kit-library convention) and the flag-injection sanitizer are fixed — the underlying design direction isn't in question, just the current implementation's verifiability.

…te, driver, egress, timeout)

Resolve the six findings from the AI-assisted 5-role adversarial PR review
(all confirmed against the diff). Design direction was affirmed; these are
implementation hardening.

1. Fail-silent -> observable (finding #1). The install now writes a durable
   success marker /var/lib/acq/oci-engine-ready (state=configured on a real
   install; state=base-engine on the functional-engine early-out). The
   fail-soft paths deliberately do NOT write it, so an operator/CI can tell a
   provisioned sandbox from one where OCI setup fell soft. scripts/verify
   checks the marker first.

2. Package-manager flag injection (finding #2). The OCI_ENGINE_PKGS sanitizer
   allowed a leading '-' token, so "podman --allow-unauthenticated" (apt) /
   --nogpgcheck (dnf) / --allow-untrusted (apk) word-split into the elevated
   install and disabled signature verification. Now validated token-by-token,
   rejecting any leading-'-' or non-word-safe token, failing closed.

3. Unconditional device grant (finding #3). oci-engine-grant-devs.sh now GATES
   on `command -v podman`: no engine, no chown/chmod of /dev/net/tun + /dev/fuse
   (avoids needlessly widening egress-relevant device access for a capability
   that does not exist).

4. Storage driver mis-pinned from a boot-transient probe (finding #4). Driver
   selection moved to a shared helper (oci-engine-config/oci-engine-storage-driver.sh)
   sourced by both the create install and the every-boot grant step, converging
   vfs -> overlay once fuse-overlayfs + /dev/fuse are present. Only rewrites a
   kit-owned storage.conf (sentinel), never an operator's.

5. Missing caps.network for the registry (finding #5). spec.yaml now declares
   the Docker Hub registry/auth/blob-CDN hosts the kit's purpose needs, so
   runtime pulls work even on a strict tier. The distro package mirror stays
   with the balanced baseline (distro-specific, already in core).

6. Unbounded boot-path self-test (finding #6). The per-boot rootless
   `podman build` self-test now runs under `timeout`
   (OCI_ENGINE_SELFTEST_TIMEOUT, default 120s) so a wedged mount cannot hang
   the boot; a timeout counts as a failed self-test and triggers the vfs
   fallback.

Also folds in the verify base-image clarification (engine-free :shell base;
`*-docker` variants run dockerd and would trip the early-out). spec.yaml,
README, and docs/decisions updated; validator/sh -n/scan-shell clean; sanitizer,
marker, and storage-convergence paths verified with stubbed shells.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul

mogul commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough adversarial pass — all six findings addressed in 0d06d08. Summary of what changed (design direction unchanged; this is implementation hardening):

1. Fail-silent → observable (marker + verify). The install now writes a durable success marker /var/lib/acq/oci-engine-ready — state=configured on a real install, state=base-engine on the deliberate functional-engine early-out. The fail-soft paths deliberately do not write it, so an operator/CI can distinguish a genuinely-provisioned sandbox from one where OCI setup fell soft. scripts/verify (the per-kit convention this kit already follows) now checks the marker first and dumps the create log’s oci-engine lines on a miss. (The kit ships scripts/verify; it was in the branch but not yet in the reviewed tree.)

2. Package-manager flag injection. Correct catch. The sanitizer permitted a leading - token, so OCI_ENGINE_PKGS="podman --allow-unauthenticated" (apt) / --nogpgcheck (dnf) / --allow-untrusted (apk) word-split into the elevated install. It now validates token-by-token and rejects any leading-- or non-word-safe token, failing closed (skip OCI setup). Verified with a stub that records apt-get argv: the injected flag is refused and apt-get is never called; a legit set still passes through.

3. Unconditional device grant. oci-engine-grant-devs.sh now gates on command -v podman — no engine, no chown/chmod of /dev/net/tun + /dev/fuse. Agreed on the /dev/net/tun egress relevance.

4. Storage driver mis-pinned from a boot-transient probe. Driver selection moved into a shared helper (oci-engine-config/oci-engine-storage-driver.sh) sourced by both the create-time install and the every-boot grant step. It converges vfs → overlay once fuse-overlayfs + /dev/fuse are both present, and only ever rewrites a kit-owned storage.conf (sentinel comment), never an operator’s. Verified: no-fuse → vfs, then fuse appears → upgraded to overlay; operator file left untouched.

5. Missing caps.network egress. Added. The kit’s purpose (image pulls) needs Docker Hub, which — unlike the distro package mirror — is not distro-dependent, so spec.yaml now declares registry-1.docker.io, auth.docker.io, index.docker.io, and the production.cloudflare/cloudfront.docker.com blob CDNs. Runtime pulls now work even on a strict tier. The distro package mirror is intentionally still left to the balanced baseline (it is distro-specific and already in the baseline core set); the spec documents that split explicitly.

6. Unbounded self-test on the boot path. The per-boot podman build self-test now runs under timeout (OCI_ENGINE_SELFTEST_TIMEOUT, default 120s); a timeout counts as a failed self-test and triggers the vfs fallback rather than hanging boot.

Validation: validate-kits --strict passes for all 8 kits; sh -n + bash -n clean on every script; scan_unsafe_shell 0/0. The kit’s docs/decisions records the rationale for each change (fail-soft-with-marker, sanitizer, podman-gate, driver convergence, registry egress, self-test timeout).

Live end-to-end run on an engine-free msb base (docker/sandbox-templates:shell, adapter auto-install off) previously passed 9/9; will re-run to confirm the marker + driver-convergence paths on hardware before marking ready for review.

@mogul

mogul commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Live end-to-end re-run on hardware (engine-free msb base docker/sandbox-templates:shell, adapter auto-install off) — 10/10 pass, including the new review-driven checks:

3. Kit success marker + podman installed (as the agent)
  PASS kit success marker present (/var/lib/acq/oci-engine-ready)
        installed_at=2026-09-23T00:40:05Z;state=configured;podman=/usr/bin/podman;docker_wrapper=/usr/local/bin/docker;short_name_mode=enforcing
  PASS podman present (version: 5.7.0)
4. rootless podman info succeeds as the agent
  PASS podman info succeeds as the agent
  PASS podman reports rootless=true
5. real layer mount: podman build FROM scratch as the agent (rootless)
  PASS podman build FROM scratch succeeded (rootless layer mount works)
6. docker -> podman wrapper (dropped by THIS kit) resolves on PATH
  PASS docker resolves to the kit wrapper (/usr/local/bin/docker)
  PASS docker wrapper routes to podman (reported: 5.7.0)

Summary
  10 passed, 0 failed

This exercises the finding #1 marker (state=configured), the podman-gated device grant + storage-driver helper (#3/#4) producing a working rootless engine (rootless=true, real layer-mount build), and the kit-dropped docker->podman wrapper. All six findings are addressed in 0d06d08 and confirmed live.

@wz-gsa — all six findings from the adversarial panel are resolved and verified live; could you re-run the review pass when you have a chance? Happy to iterate on anything else it surfaces.

…012)

The docs/decisions note ended with two consecutive blank lines, tripping
MD012/no-multiple-blanks in the Markdown Lint CI check. Trim to a single
trailing newline.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul
mogul marked this pull request as ready for review September 23, 2026 05:51
@wz-gsa

wz-gsa commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

AI-assisted adversarial review (OpenCode), advisory — needs a human to confirm before it drives a change. Reviewed at head 429b0b1, main at 5d1a877. Independent capacity, reviewing alongside the other session rather than duplicating it.

I read the six findings from the 09-22 panel and confirmed all six are genuinely fixed at this head — details at the bottom, because a re-review that only reports new problems isn't much use. One new finding, which I'd treat as blocking, and it's specific to a kit whose whole job is running privileged setup.

Blocker: root sources a helper from an agent-writable path, on every boot

spec.yaml:110-113 drops the shared storage-driver helper into the agent's home:

- path: /home/agent/oci-engine-config/oci-engine-storage-driver.sh
  mode: "0644"

Two elevated steps then source it:

oci-engine-install.sh:175      . "$STAGE_DIR/oci-engine-storage-driver.sh"   # create-time, user: "0"
oci-engine-grant-devs.sh:96    . "$STAGE_DIR/oci-engine-storage-driver.sh"   # EVERY START, user: "0"

STAGE_DIR is /home/agent/oci-engine-config — inside the agent's own home, materialised as the agent user. . in POSIX sh executes the file's contents in the calling shell, so whatever is in that file at start time runs as uid 0.

The sandbox's own threat model makes this reachable: the agent has passwordless sudo, and quickstart's ADR-0030 states the consequence directly — "inside the guest, root-ownership is not a trust boundary against the agent." But this doesn't even need sudo. The file is in the agent's home at mode 0644, owned by the agent. A prompt-injected agent appends one line and waits for the next acq start, at which point the kit hands it root — no privilege escalation step required, because the kit performs it.

The create-time call is less interesting (nothing has run yet). The startup call at grant-devs.sh:96 is the problem: it re-executes on every boot, after the agent has had a full session to write to its own home.

Worth noting this kit is the reason quickstart #504 built Mechanism 2 — readonly: true staging on the host, mounted :ro, with the startup argv rewritten to the read-only path, precisely so trusted startup code can't be tampered between restarts. No file in this spec declares readonly: true. Options, roughly in order of preference:

  1. Declare the helper readonly: true and let #504's :ro staging own it (couples this kit to that PR landing).
  2. Have the root install step copy the helper to a root-owned path (/usr/local/lib/acq/ at 0644 root:root) and have the startup step source that, not the staging copy — same pattern the kit already uses for docker → /usr/local/bin/docker and the registries conf → /etc/containers/registries.conf.d/.
  3. Inline the helper into both scripts and drop the shared file.

(2) is the smallest change and matches what this kit already does correctly for its other two staged files.

Should-fix: the two elevated scripts are themselves agent-writable

Same mechanism, one step removed. spec.yaml:88-105 drops oci-engine-install.sh and oci-engine-grant-devs.sh into /home/agent/ at 0755, and the spec invokes them as sh /home/agent/oci-engine-grant-devs.sh with user: "0". The startup one re-reads from that path every boot.

This is arguably the kit vocabulary's problem rather than this kit's — every kit that declares a root startup command inherits it, and the sibling kits do the same thing. But this is the kit with the most root surface (two elevated phases, device-permission changes, a package install), so it's where the pattern is worth naming. If option (2) above is taken for the helper, the same argument applies to these two.

Nit: /dev/net/tun stays granted after podman disappears

The command -v podman gate (fixing panel finding #3) is the right call and I confirmed it's there — grant-devs.sh:45-48 exits 0 before any grant. But the grant isn't revoked: if podman is present at boot N and absent at boot N+1 (a base image change, a failed re-install), the node keeps root:agent 0660 from the earlier boot, because nothing walks it back. The gate prevents new grants, not stale ones. Given the script's own comment calls /dev/net/tun "egress-relevant — it enables in-guest tunnelling", a symmetric revoke on the no-podman path would match the stated intent.

Nit: egress declares Docker Hub but the install needs distro mirrors

README.md:103-124 explains this deliberately: the registry hosts are declared because they're distro-independent, while the package mirror is left to the balanced baseline because its host isn't. The reasoning is sound and I'm not asking for hardcoded mirrors. But the stated consequence — under ACQ_NETWORK_TIER=strict "the install fails soft (recorded by the absence of the success marker)" — means the kit's headline capability silently doesn't exist on a strict tier, and the only signal is a missing marker. Given the panel's finding #1 was exactly this class, it'd be worth having scripts/verify assert the marker explicitly so a strict-tier run reports why it's degraded rather than just being quiet.

Panel findings #1-#6: all six confirmed fixed at 429b0b1

  • Epic: Repository Bootstrap #1 fail-silent — scripts/verify now exists (the panel noted its absence), and the offline/live split is real.
  • Complete remaining templates #2 flag injection — fixed properly at oci-engine-install.sh:63-79: a per-token loop rejecting a leading - and an unsafe charset, failing closed. The comment names --allow-unauthenticated / --nogpgcheck / --allow-untrusted explicitly and explains why the charset check alone was insufficient. That's the right fix and the right documentation of it.
  • Create starter skills (4 patterns) #3 unconditional device grant — grant-devs.sh:45-48 now gates on command -v podman and exits 0 with a message. (See my nit above about revocation.)
  • Create starter prompts (3 patterns) #4 storage driver mis-pinned from a boot-transient probe — the shared helper is now re-evaluated on every start, converging vfs→overlay. The fix is real; it's also what created the blocker above.
  • Create starter agent patterns (3) #5 missing caps.network — now declared, 5 exact Docker Hub hosts, no wildcards.
  • Create starter workflows (2) #6 unbounded podman build — the self-test moved into the gated vfs-fallback path.
  • Also verified: OCI_ENGINE_AGENT_USER is charset-guarded before it reaches chown (grant-devs.sh:59-69) and falls back to agent rather than failing open; OCI_ENGINE_SHORT_NAME_MODE validates against a three-value allowlist; the docker shim is exec podman "$@" with no sudo, so the engine really is rootless; device grants are group-scoped 0660, not 0666.

Method: static analysis at 429b0b1. No kit was executed, no sandbox created, no podman run. The blocker is read from the three cited lines plus POSIX . semantics; I did not demonstrate it in a live sandbox. UNVERIFIED: whether the file-drop mechanism on either backend happens to land these files root-owned despite the spec dropping them as the agent — if it does, the blocker weakens to the staging-copy case only, and that would be worth stating in the spec rather than left implicit. A 5-voter pr_review panel was attempted for this and the sibling #438; all voters errored (JSON parse failures, producerVersion 8.81.2), so this is single-reviewer and I'm saying so rather than implying panel weight.

Also recording, so a scanner result is not left unexplained: bin/injection-scan.sh flagged one hit on this diff — a (system|assistant|developer): pattern match on README.md:61, which is prose about the overlay filesystem (the sandbox's **overlay root** filesystem). Inspected, benign, no injection attempt anywhere in the diff.

Move elevated OCI kit execution through root-owned copies under /usr/local/lib/acq/oci-engine and verify the trusted path ownership. Also revoke stale device grants when podman is absent.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul

mogul commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor Author

AI-assisted follow-up addressing the latest review at 429b0b1. Changes landed in e38be4d.

Summary of changes since the review:

  • Fixed the blocker: root phases no longer execute or source agent-writable staging files under /home/agent.
  • The install command now first publishes /home/agent/oci-engine-install.sh to /usr/local/lib/acq/oci-engine/oci-engine-install.sh as root-owned code, then executes that trusted copy.
  • The install script now publishes the startup grant script and storage-driver helper into /usr/local/lib/acq/oci-engine/ as root-owned files before later lifecycle use.
  • The startup command now executes /usr/local/lib/acq/oci-engine/oci-engine-grant-devs.sh, and that script sources only /usr/local/lib/acq/oci-engine/oci-engine-storage-driver.sh.
  • Addressed the stale-grant nit: when podman is absent, the startup script best-effort resets /dev/net/tun and /dev/fuse to root:root 0600 before exiting.
  • Strengthened scripts/verify to assert /usr/local/lib/acq, /usr/local/lib/acq/oci-engine, and the trusted install/startup/helper files are root-owned with expected modes.
  • Updated README and the decision note to document the trusted root-code handoff and stale device-grant behavior.

Verification run locally:

sh -n integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-install.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-grant-devs.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-vfs-fallback.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-config/oci-engine-storage-driver.sh
PASS

bash -n integrations/isolation/acq-kits/oci-engine/scripts/verify
PASS

shellcheck -S warning integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-install.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-grant-devs.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-vfs-fallback.sh \
  integrations/isolation/acq-kits/oci-engine/files/home/oci-engine-config/oci-engine-storage-driver.sh \
  integrations/isolation/acq-kits/oci-engine/scripts/verify
PASS

git diff --check
PASS

PATH="$PWD/.venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=1 make validate-kits
PASS: All 8 acq-kits valid.

PATH="$PWD/.venv/bin:$PATH" make validate
PASS: all validators passed; existing advisory sensitive-term warnings only.

PATH="$PWD/.venv/bin:$PATH" make scan-shell
PASS: unsafe-shell scan: 0 error(s), 0 warning(s)

PATH="$PWD/.venv/bin:$PATH" make test
PASS: 401 passed

Host-side live verification supplied by Bret from integrations/isolation/acq-kits/oci-engine:

scripts/verify

0. Preconditions
  acq present; msb backend responds.
  PASS kit spec.yaml present

1. Backend-agnostic gate (validate-kits) sees this kit
  PASS spec.yaml is the oci-engine hybrid/v1 mixin

2. Create an msb sandbox with the kit applied (adapter OCI auto-install OFF)
  base image: docker.io/docker/sandbox-templates:shell (engine-free: docker CLI w/ dead socket, no podman)
  creating shell-oci-engine-verify-fhgc1a (workspace /Users/bretamogilefsky/.oci-engine-verify.fHGC1A; ACQ_MSB_ENSURE_OCI=0)...
  PASS sandbox created via acq with --kit

3. Kit success marker + podman installed (as the agent)
  PASS kit success marker present (/var/lib/acq/oci-engine-ready)
        installed_at=2026-09-24T02:10:39Z;state=configured;podman=/usr/bin/podman;docker_wrapper=/usr/local/bin/docker;short_name_mode=enforcing
  PASS podman present (version: 5.7.0)

4. Trusted root startup payloads are root-owned
  PASS trusted startup path and payloads are root-owned under /usr/local/lib/acq/oci-engine
        0:0 755 /usr/local/lib/acq;0:0 755 /usr/local/lib/acq/oci-engine;0:0 755 /usr/local/lib/acq/oci-engine/oci-engine-install.sh;0:0 755 /usr/local/lib/acq/oci-engine/oci-engine-grant-devs.sh;0:0 644 /usr/local/lib/acq/oci-engine/oci-engine-storage-driver.sh

5. rootless podman info succeeds as the agent
  PASS podman info succeeds as the agent
  PASS podman reports rootless=true

6. real layer mount: podman build FROM scratch as the agent (rootless)
  PASS podman build FROM scratch succeeded (rootless layer mount works)

7. docker -> podman wrapper (dropped by THIS kit) resolves on PATH
  PASS docker resolves to the kit wrapper (/usr/local/bin/docker)
  PASS docker wrapper routes to podman (reported: 5.7.0)

Summary
  11 passed, 0 failed
  All checks passed.

@wz-gsa please re-review the root-owned execution path and stale-grant handling when you have a chance.

@wz-gsa

wz-gsa commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

AI-assisted re-review (OpenCode), advisory — needs a human to confirm before it drives a change. Reviewed current head e38be4d/28d05eb.

This re-confirms the second review's own findings, then independently re-verifies the ORIGINAL panel's 6 findings at the current head too — since a root-hardening commit that fixes a new blocker doesn't automatically mean the earlier fixes are still intact, and I wanted to check rather than inherit that on trust. Ran two passes: (1) direct file reads + a live install ownership test for the blocker, (2) a fresh subagent re-deriving each of the original 6 findings from the current code, independent of any changelog claim.

Second review's blocker + 2 nits: confirmed fixed

Blocker (root sourcing a helper from an agent-writable path) — fixed correctly. oci-engine-install.sh now publishes both the storage-driver helper and the grant script into a root-owned /usr/local/lib/acq/oci-engine/ via install -m ... before any fail-soft exit. I live-tested the underlying assumption: ran a real install -m 0755 <agent-owned-src> <dst> as a non-root test and confirmed POSIX install produces a destination owned by the calling uid regardless of source ownership — so root running install genuinely produces a root-owned file. Confirmed via grep that no root-phase command in spec.yaml or any of the 3 shell scripts sources/execs anything under /home/agent/ anymore.

Nit 1 (stale device grant not revoked) — on rereading, this was already handled: the podman-absent branch in grant-devs.sh actively reverts /dev/net/tun and /dev/fuse back to root:root 0600, not just skips granting.

Nit 2 (verify doesn't assert the marker) — also already handled: scripts/verify explicitly checks for /var/lib/acq/oci-engine-ready and reports "ABSENT — the install fell soft" as a distinct failure mode.

Original panel's 6 findings, re-verified fresh at the current head (not inherited from the changelog)

# Finding Verdict
1 Fail-silent, not fail-soft Confirmed fixed — traced every exit path in oci-engine-install.sh; the marker (state=configured) is only reachable past every fail-soft exit 0, and scripts/verify now does a real live check (creates a real sandbox, runs an actual podman build, checks rootless=true)
2 Package-manager flag injection Confirmed fixed — traced OCI_ENGINE_PKGS="podman --allow-unauthenticated" through the current validation: the -* case matches the second token before the charset check runs, sets the bad-flag, hits exit 0
3 Unconditional device grant Confirmed fixed — the podman-absent branch now gates AND self-heals (reverts prior grants back to root:root 0600)
4 Storage driver permanently mis-pinned Confirmed fixed — the helper re-probes /dev/fuse on every call and is now invoked from both create-time AND every-boot (grant-devs.sh), not just create-time
5 Missing caps.network egress Confirmed fixed — 5 explicit Docker Hub hosts, no wildcards
6 Unbounded podman build on boot Partially fixed — a timeout wrapper now exists (default 120s, OCI_ENGINE_SELFTEST_TIMEOUT-overridable), applied to both build calls in oci-engine-vfs-fallback.sh. But the code's own comment documents that when no timeout binary is present on the base image, it falls back to running the build unbounded rather than skipping/failing — so the original defect (a wedged storage lock or hung newuidmap stalling every boot indefinitely) persists on any base lacking coreutils' timeout applet. This runs on every sandbox start, not just install, so it's worth closing rather than leaving as a known gap.

Recommendation

5 of 6 original findings plus the blocker and both nits from the second review are genuinely fixed, verified by direct code reading and a live test — not by trusting the PR's own summary. I'd lean toward approve, with one non-blocking ask: either fail closed (skip the self-test, not run it unbounded) when timeout is unavailable, or open a tracked follow-up issue committing to that fix before this kit sees broader rollout. This is a real, currently-accepted gap on the boot path, not a hypothetical.

Given how much this PR has already tightened across three rounds of review, I don't think #6 alone should block merge — flagging it as the one thing I'd want a decision on (fix now vs. tracked follow-up) rather than silently letting it ride.

Skip the boot-path rootless podman build self-test when timeout is unavailable, and normalize invalid or zero timeout overrides back to the default so operators cannot accidentally disable the bound.

Add regression coverage for missing timeout and invalid timeout values.

Co-authored-by: OpenCode Agent <bret.mogilefsky@gsa.gov>
@mogul

mogul commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

AI-assisted update pushed for re-review. Human review required before merge.

Summary:

  • Updated the PR branch from origin/main with a non-destructive merge commit.
  • Closed the latest wz-gsa re-review item by making the boot-path rootless podman self-test require a bounded timeout.
  • If timeout is unavailable, oci-engine-vfs-fallback.sh now skips the self-test/fallback path instead of running podman build unbounded on every boot.
  • Invalid or zero timeout overrides are normalized to the default 120s, including '', 0, 00, 000, and non-numeric values.
  • Added pytest regression coverage for missing timeout and invalid timeout normalization.
  • Updated README and the decision note to document the fail-closed/skip posture.

Adversarial review:

  • Ran a read-only adversarial sub-agent review focused on the timeout patch. It found the leading-zero zero-timeout bypass; fixed it.
  • Ran a final adversarial sub-agent review after the fix; no findings. Residual risk: live sandbox boot-path validation still needs a host with acq/msb available.

Verification transcript:

  • git fetch origin main feat/oci-engine-kit: PASS
  • git merge --no-edit origin/main: content merge applied; merge commit initially blocked by local git SSH signing socket, then completed with commit.gpgsign=false for that commit only: PASS
  • sh -n oci-engine shell scripts: PASS
  • bash -n integrations/isolation/acq-kits/oci-engine/scripts/verify: PASS
  • shellcheck -S warning on oci-engine shell scripts + scripts/verify: PASS
  • PYTHONDONTWRITEBYTECODE=1 uv run --extra dev make validate-kits: PASS, all 9 acq-kits valid
  • PYTHONDONTWRITEBYTECODE=1 uv run --extra dev make validate: PASS, existing advisory sensitive-term/network warnings only
  • PYTHONDONTWRITEBYTECODE=1 uv run --extra dev make scan-shell: PASS, 0 errors / 0 warnings
  • PYTHONDONTWRITEBYTECODE=1 uv run --extra dev pytest scripts/tests/test_oci_engine_vfs_fallback.py -v: PASS, 6 passed
  • PYTHONDONTWRITEBYTECODE=1 uv run --extra dev make test: PASS, 407 passed
  • ruff check + ruff format --check on the new test: PASS
  • git diff --check: PASS
  • integrations/isolation/acq-kits/oci-engine/scripts/verify: BLOCKED locally because acq is not on PATH; no live msb backend check could run from this worktree
  • pre-commit run --all-files: BLOCKED locally by environment disk exhaustion while compiling the gitleaks hook (Go linker: no space left on device). Branch-caused equivalents above passed; hosted CI should run this in a clean environment.

Please re-review the bounded self-test behavior and timeout normalization when available.

@mogul
mogul merged commit b980c93 into main Oct 2, 2026
7 checks passed
@mogul
mogul deleted the feat/oci-engine-kit branch October 2, 2026 07:41
wz-gsa added a commit that referenced this pull request Oct 2, 2026
…o converted

CI caught a real defect in the contract test, from a direction I had not
anticipated. Rebased onto main (1.10.0) and fixed.

WHAT HAPPENED. `oci-engine` (#437) landed on main while this change was in
review — an EIGHTH verify script that did not exist when I branched. The test
derived its converted set by subtraction:

    {k: p for k, p in _verify_scripts().items() if k not in UNCONVERTED}

so a kit in neither set was treated as CONVERTED by default. The four contract
tests then failed against it with "oci-engine/scripts/verify does not source
lib/verify-report.sh", which is true but misleading: the real problem was that a
kit appeared that nobody had classified.

I had guarded both directions I thought of — a kit converted but still listed,
and a listed kit that reverted — and missed the one that actually happened. The
default was the bug: an unclassified kit silently inherited a contract it does
not implement, which is the same shape as the defect this whole change exists to
fix (something unmeasured treated as passing).

THE FIX. Two explicit sets, CONVERTED and UNCONVERTED, plus
`test_every_kit_is_classified` asserting that every verify script appears in
exactly one of them. A kit in neither is now a hard error naming the real cause,
and the sets cannot name a kit that has no verify script.

Proved it by simulating a ninth unclassified kit:

    AssertionError: verify script(s) in no classification set: ['zz-phantom'].
    Add each to CONVERTED (if it sources verify-report.sh) or to UNCONVERTED
    (if it is still on the old two-state verdict). A kit is never converted
    by default.

CORRECTED COUNTS, recounted on the rebased tree rather than carried forward:
there are EIGHT verify scripts, not seven, and 35 uncounted `skip`/NOTE sites,
not 29 (9 `warn` sites unchanged). oci-engine itself has 303 lines, 2 gates, and
2 `[ "$fail" -eq 0 ]` verdicts, so it has the primary defect too; it is listed as
UNCONVERTED for the staged bulk conversion. Stale numbers in the library header
and the test docstring are updated.

UNCONVERTED is now: agentic-coding-playbook, oci-engine, openchamber, paseo,
pi-coding-agent, zscaler-ca-certificate. It must reach empty.

Unrelated to the code, but it is why this took a second pass: the CI log could
not be read at first because `gh run view --log` failed with
`x509: certificate signed by unknown authority`. Root cause is Go-specific and
documented in the review workspace, not here — Go's x509.SystemCertPool() omits
macOS admin-installed roots (121 of the host's 177), so any Go tool fails against
a proxy-intercepted host while `curl` to the same host verifies fine. Reading the
log required pointing SSL_CERT_FILE at a bundle including the admin roots. No
repo change needed; noted so the next person who sees that error does not mistake
it for a CI outage.

Verified on the rebased tree:

    pytest scripts/tests/                     -> 420 passed (13 contract tests)
    unclassified-kit probe                    -> FAILS with the naming assertion
    validate_repo.py --root .                 -> exit 0
    validate-kits.py --root . --strict        -> all kits OK
    scan_unsafe_shell.py .                    -> 0 errors, 0 warnings
    shellcheck -s sh verify-report.sh         -> exit 0
    ruff check / ruff format --check          -> clean

AI assistance disclosed: prepared by an AI agent (OpenCode) on operator
instruction. Human review and merge still required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants