bug(gateway): finalized ephemeral sandboxes remain Completed while supervisor stays connected

[johntmyers]

User Story

As an operator running short-lived agent jobs, I want an ephemeral sandbox to be removed after its main process finishes and its terminal output has been delivered, so that completed jobs do not leave compute resources running.

Problem Statement

A sandbox created with --no-keep --detach receives the openshell.nvidia.com/retention: ephemeral annotation, but remains in Completed after its canonical process exits successfully. Its supervisor stays connected to retain the control-mode access plane, so the sandbox and its Docker containers remain present indefinitely until manually deleted.

Impact / Why This Matters

Unattended job launchers such as Gator can accumulate completed sandboxes and running supervisor/compute containers despite requesting self-deletion. The current workaround is to list and manually delete completed ephemeral sandboxes, which requires ongoing operator attention and does not prevent resource buildup between cleanups.

Acceptance Criteria

• After the canonical process result and terminal delivery are finalized, an ephemeral sandbox is deleted even while its supervisor session remains connected.
• Cleanup does not begin before terminal output delivery is finalized.
• Retained sandboxes remain available after their canonical process exits.
• A gateway regression test covers a finalized ephemeral sandbox with a still-connected supervisor session and verifies that cleanup starts without a disconnect.
• A real end-to-end test runs --no-keep --detach against the Docker, Podman, Kubernetes, and VM driver lanes, then verifies that the sandbox record and driver resources disappear without a CLI delete. Cover both successful and failed canonical-process exits. Add equivalent coverage to the separate MXC e2e lane where that driver supports this lifecycle.

Reproduction Steps

1. Start the local Docker gateway (mise run gateway:docker).
2. Run openshell --gateway docker-dev sandbox create --name eph-check --no-keep --detach -- /bin/true.
3. Run openshell --gateway docker-dev sandbox get eph-check -o json and docker ps after the command exits.
4. Observe phase: Completed, exit_code: 0, and openshell.nvidia.com/retention: ephemeral. The sandbox and supervisor containers remain running. The supervisor logs Canonical process exited; retaining control-mode access plane.
5. Manually delete the sandbox with openshell --gateway docker-dev sandbox delete eph-check.

Environment

• OpenShell: 0.1.3-dev.28+g798500ccd
• OS: macOS 26.7, arm64
• Runtime: Docker Desktop engine 29.7.2; local docker-dev gateway
• Reproduced on 2026-09-29 local time

Logs

$ openshell --gateway docker-dev sandbox list
NAME       CREATED              PHASE
eph-check  2026-09-30 06:20:08  Completed

supervisor: Canonical process exited; retaining control-mode access plane

Agent Investigation

The gateway schedules ephemeral deletion only in the supervisor-session disconnect path when terminal_delivery_finalized is true (crates/openshell-server/src/compute/mod.rs, around line 4456). The supervisor intentionally retains control-mode access after reporting and finalizing the process exit (crates/openshell-supervisor/src/lib.rs, around line 1293). The existing test ephemeral_cleanup_waits_for_terminal_finalization explicitly waits for a disconnect before expecting deletion (crates/openshell-server/src/compute/mod.rs, around line 9265). The finalize RPC records terminal delivery while the session is still connected (crates/openshell-server/src/supervisor_session.rs, around line 2055).

The shared Rust lifecycle e2e suite already tests attached --no-keep commands, but attached CLI cleanup can mask the gateway bug (e2e/rust/tests/sandbox_lifecycle.rs, around line 1345). Its detached canonical-process tests use persistent sandboxes (e2e/rust/tests/sandbox_lifecycle.rs, around line 760). Extend that shared suite with detached ephemeral cleanup and run it in each applicable driver lane; MXC has a separate Windows e2e runner.

[johntmyers]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High on the gateway cause; Medium on cross-driver execution

Summary

After the gateway validates and finalizes an ephemeral sandbox's terminal result, start the existing deletion path without waiting for its supervisor session to disconnect. Keep disconnect-triggered cleanup as a fallback for ordering races. Add unit and detached end-to-end coverage that cannot be satisfied by the CLI's attached-session cleanup.

Scope

• crates/openshell-server/src/supervisor_session.rs and crates/openshell-server/src/compute/mod.rs: trigger deletion after finalization/session bookkeeping for a terminal ephemeral result, preserving instance validation, restart transitions, and retained sandboxes.
• Gateway unit tests: require deletion while the session remains connected; cover the before-finalize boundary and terminal/restart/retained cases.
• Shared Rust e2e lifecycle test and lane wrappers: create detached --no-keep sandboxes with successful and failed commands; verify the gateway record and compute resources disappear without a CLI delete in Docker, Podman, Kubernetes, and VM lanes. Ensure the Podman CI selection and VM wrapper actually run the test.

Implementation Steps

1. Add an idempotent gateway cleanup trigger for a validated, finalized terminal ephemeral sandbox. Invoke it after session finalization; retain the disconnect fallback. Verify unit cases for ordering and scope.
2. Add a driver-neutral detached ephemeral e2e scenario for success and failure, with driver resource checks and failure-only test cleanup.
3. Wire the scenario into the four applicable driver lanes, run focused tests and repository checks, then open a PR.

Test Plan

• Unit tests: No deletion on exit report alone; deletion after finalization while connected; successful and failed terminal results; retained and restarting sandboxes remain; duplicate/fallback events are harmless.
• Integration tests: Gateway finalization RPC ordering and session bookkeeping, where existing test harness supports it.
• E2E tests: Detached --no-keep success and failure in Docker, Podman, Kubernetes, and VM, checking both gateway absence and driver resource removal. Run available lanes locally and report any environment-limited lanes precisely.

Risks & Open Questions

• Deleting inside compute.finalize_main_process_exit would race the session registry mark and could make the finalize RPC fail. The trigger belongs after the registry mark, with a safe fallback if the session disappears between those operations.
• MXC does not use ConnectSupervisor or the supervisor finalization RPC. This gateway trigger cannot exercise MXC; equivalent MXC cleanup coverage requires its separate lifecycle path and is outside this fix.
• The test must not call the CLI's attached cleanup or manually delete before its assertions.

Documentation Impact

No gateway TOML, driver options, Helm rendering, public API, or documented workflow changes. This restores the existing --no-keep promise. No LSM-sensitive process identity or /proc behavior changes.

---

Revision 1 — initial plan

[johntmyers]

🏗️ build-from-issue-agent

Implementation Complete

PR: #3984

The gateway starts ephemeral driver deletion once the canonical main-process result is finalized, even while the supervisor control session remains connected. Existing disconnect cleanup remains as a fallback.

Three focused unit tests cover connected cleanup, failure cleanup, and retained/restarting exclusions. Two shared detached e2e tests check both gateway record deletion and driver resource removal across the supervisor-based driver suites. Local Docker execution passed both tests and all six conformance scenarios. Final full local CI and pre-commit checks passed; the earlier unrelated supervisor-network timeout did not recur in the final CI run. Live Podman, Kubernetes, and VM verification remains unperformed locally.

Updated the Gator skill's cleanup description; the published sandbox docs already describe the intended behavior. This issue will close when the PR merges.