[Enrique Saurez (esaurez)]

host::instance, which aci_edge_sandboxes drives, kept its own runtime record, launch record, lock, and spawn, wait, and kill code. This PR moves it onto the host layers that the CLI uses since #772: host::store, host::supervisor, and host::session, under a runtime identity that the install metadata of the artifacts names. It then removes the host::process helpers that only the old code used.

What changes

• Instance. An Instance is a sandbox store in a directory of its own. Instance::create and Instance::open replace Instance::new, and Instance::lock takes the store's lock, sandbox.lock, and returns a LockedInstance:

  • find reattaches to the recorded VM with supervisor::reattach, as an orchestrator does after it restarts within the same boot: it ends a start whose caller died, forgets a VM that exited, and returns a live VM as Found::Running, which holds its capability, or the error of reading it, and a VM of another runtime identity as Found::Mismatched, which names the parts that differ and is never driven. It asks for the caller's runtime identity only if the store records a VM or a launch.
  • start launches OpenVMM through supervisor::launch with a fresh capability, under the caller's RuntimeIdentity, waits until the guest answers with the required features and has mounted the mapped paths, and commits the launch; a failed start ends OpenVMM and abandons the launch.
  • stop asks the guest to shut down through the session, and takes the supervisor's forced step after the timeout, as before, or at once when no session can reach the guest, as when the VM's capability cannot be read. Only control sessions need the capability: without it, exec fails as reading it failed, and the VM still stops.

  Instance::exec runs a workload of a Running VM without the lock, as before. instance::check_launch checks that a launch fits OpenVMM's command line before anything is recorded.

• Runtime identity. Artifacts located in a release or a checkout keep its ArtifactDir, and Artifacts::runtime_identity reads their identity from its install metadata with nvx-rs: prepare the CLI's move onto the store and supervisor #675's reader: a release's SHA256SUMS, or a checkout's build provenance. Nothing hashes an artifact. Artifacts named by path have no identity, and neither do located artifacts whose paths were changed.

• aci_edge_sandboxes. Each sandbox's state directory is its instance's store, which holds the sandbox's sandbox.json beside the store's files.

  • The store's sandbox.lock replaces the state root's .locks/ directory. A call with a stale ID still leaves nothing behind.
  • The runtime identity comes from the install metadata of the configured artifacts, or from the new OpenVmmConfig::runtime_identity, which artifacts named by path (OpenVmmConfig::new, or NVX_OPENVMM, NVX_KERNEL, and NVX_INITRD) need. Without one, probe, provision, and start fail with backend_unavailable.
  • Every operation finds the VM through LockedInstance::find. A VM of another runtime identity fails every operation with backend_error and keeps running; stopping it is left to the policy of nvx-rs: host cold integration for image-backed sandboxes #622.
  • The real-hypervisor test locates its artifacts in the checkout, whose build provenance identifies them, instead of naming them by path.

• Removed: instance's STATE_FORMAT, RuntimeRecord, LaunchRecord, ProcessIdentity, lock, create_private_dir, reconcile, and launch recovery; and host::process::{spawn, detach_child, kill_child, wait_for_exit, kill, open_identity, open_identity_with} and its re-export of supervisor::Launched.

Format switch

Switching to this build needs one stop of every running sandbox of aci_edge_sandboxes, with the build that started it. Nothing is migrated: the store refuses the runtime records and launch journals of instance's format 3, in every operation, with a message that names the format and asks to stop the sandbox with the build that started it. sandbox.json keeps its format 3, so stopped sandboxes start with this build. The earlier build's lock files stay in <state_root>/.locks/, which this build ignores; delete that directory once no earlier build runs. The two builds take different locks, so don't run them on one state root at once.

Tests

• nvx-rs/tests/fake_openvmm.rs, which drives Instance end to end against the fake OpenVMM:
  • a restarted orchestrator, in another process, reattaches to a live VM and runs a workload in it; the same VM is reported as Mismatched under another OpenVMM identity, naming that part, and is still driven, and stopped, under its own;
  • a guest that never answers a stop is terminated;
  • a VM whose capability file is missing or truncated is still found running: exec fails as reading the capability fails, and stop terminates the VM and clears its records.
• host::instance unit tests: an instance that records no VM needs no runtime identity; a runtime record or launch journal of format 3 fails, naming its format; a launch that does not fit the command line fails before the instance exists.
• host::artifacts: located artifacts have the identity of their directory, and lose it when their paths change.
• aci_edge_sandboxes:
  • tests/openvmm_fake.rs: the runtime identity of a release comes from its SHA256SUMS, a changed digest fails start, exec, stop, and deprovision with backend_error and leaves the VM running until the digest returns, and a missing SHA256SUMS fails probe with backend_unavailable; runtime records of the previous format are refused; interrupted starts are recovered through the store's journal, before and after OpenVMM's identity is recorded, and a launch whose log is released ends; a sandbox whose capability file is lost fails exec with backend_error, still reports already_started to start and deprovision, and stops with forced.
  • tests/openvmm_unavailable.rs: artifacts named by path without a runtime identity fail probe and provision with backend_unavailable.
  • src/openvmm/state.rs: stale IDs, also of an emptied directory or one holding only its lock file, and state removed while a caller waits for the lock, leave nothing behind.

Part of #620. Refs #535.

Validation

Run on Windows 11 (MSVC, WHP) and on an Azure Linux 3 host with MSHV; the earlier heads below also ran on Ubuntu in WSL2 (KVM). Neither WSL's nested KVM nor the MSHV host can present the built-in profile of its CPU (intel.alderlake.v1 and amd.genoa.v1), so OpenVMM ran there through a wrapper that adds --cpu-profile host, which cpu-profile-fallback cannot pass by design.

• At this head (a869a3e8), on dev at d4d00af8, which merged nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772. Real VMs ran with nvx-rs built from this branch and the artifacts that nvx.py download installs from the nearest development release of dev, v0.1.0-dev.3bd0c15ea785 on WHP and v0.1.0-dev.4e0689fa9b10 on MSHV, whose OpenVMM is 7ff31ed7, the gitlink of dev.
  • python scripts/nvx.py verify.
  • nvx-rs, with NVX_BUNDLE=skip, on Windows and Linux: cargo fmt --all --check; cargo clippy --locked --all-targets --all-features -- -D warnings; cargo clippy --locked --lib --no-default-features -- -D warnings; cargo test --locked --all-features, then cargo test --locked; cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings. On Linux, the other steps of check-nvx-rs: the 1.89 check, the macOS check, the artifact-bundling test, and the three configurations of tests/runtime-adapter.
  • aci_edge_sandboxes: the steps of check-aci-edge-sandboxes on Linux (fmt, clippy, tests, nvx.py test-agent-bundling, docs, the 1.89 check, the macOS check), and its fmt, clippy, tests, and docs on Windows.
  • Each commit builds on its own: cargo check --locked --all-targets --all-features in nvx-rs and aci_edge_sandboxes, on Linux.
  • On Linux, the Specula tests, the pinned shellcheck and shfmt on the scripts that check-quality lists, and the tests of validate-nvx, test_setup_azure.py included. test_sleeper_subshell_records_and_reaps_its_sleepers passes once no other run on the shared host has its sleep 611.25 processes alive, because it looks for them in every process.
  • On Windows, the Python steps of check-quality with the pinned tools of requirements-dev.txt (ruff check, ruff format --check, pyright for Linux and Windows, the Markdown link check), the PowerShell syntax check, the tests of validate-nvx, and the nvx.py --help smoke commands. Four process-containment tests of test_adversarial.py fail identically on dev without this PR.
  • WHP: python scripts/nvx.py test-microvm --scenario managed-lifecycle --scenario sandbox-lifecycle --scenario managed-exec-config, and python scripts/nvx.py test-aci-edge-sandboxes (18 tests).
  • MSHV, under sudo for /dev/mshv: the same three scenarios; and test-aci-edge-sandboxes, where 13 of 18 tests pass and the 5 that write through read-write mappings fail as they do without this PR, at be28d631, because the unprivileged workload cannot write the root-owned directories that a root test creates.
• At 445bbe64, the same patches on nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772's be28d631, on dev at 72e8ea26: the same gates, but for the Python steps on Windows, and the same real-VM runs, with the artifacts of v0.1.0-dev.72e8ea2677bd.
• At 3e5a32dc, the same patches on nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772's 0e458e5e, before nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772's last commit: the same gates and real-VM runs, with the artifacts of v0.1.0-dev.bdaed9c6a7ca on WHP and of v0.1.0-dev.8483e3757469 on MSHV. On Windows, test_adversarial.py's test_close_is_bounded_when_a_surviving_child_holds_executor_stdout also failed in some runs, as it does on dev, because the removal of its temporary directory races the exit of the child that it killed.
• At d73f87da, before the rebase, on nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772's 9951076c, on nvx-rs: prepare the CLI's move onto the store and supervisor #675's f5cff887, on dev at ed6a603f, with the same changes but for the rustdoc of LockedInstance::start, which now takes nvx-rs: refuse read-only paths inside read-write directories on every host #672's wording: the same gates, and the same real-VM runs on WHP and MSHV, with the artifacts of v0.1.0-dev.ed6a603f1e53. On WHP, test-aci-edge-sandboxes passed its 18 tests.
• At a73868d9, the same patches on nvx-rs: prepare the CLI's move onto the store and supervisor #675's 141a04b4, before nvx-rs: prepare the CLI's move onto the store and supervisor #675 was split into nvx-rs: prepare the CLI's move onto the store and supervisor #675 and nvx-rs: move the CLI's sandbox lifecycle onto the store and supervisor #772, on dev at e714acbd: the same gates and real-VM runs, with the artifacts of v0.1.0-dev.e714acbd47bf on WHP, KVM, and MSHV.
• At 89f82304, the same patches on nvx-rs: prepare the CLI's move onto the store and supervisor #675's d2193903, on dev at 4f9e7cbd, before nvx-rs: plan sandbox launches with the image format and upper #676: the same gates and real-VM runs, with the artifacts of v0.1.0-dev.610a5ca1ca71 on WHP, of v0.1.0-dev.c61080e23971 on KVM, and of v0.1.0-dev.8f2565ba41dd on MSHV. At 3ffb38df, on nvx-rs: prepare the CLI's move onto the store and supervisor #675's 773878fb, on dev at c723fed1, before nvx-rs: separate distro verification from launch planning #653: the same, with the artifacts of v0.1.0-dev.c39666f78111 on WHP and KVM and of v0.1.0-dev.c723fed17bd1 on MSHV.
• At 7ece0b56, the same patches on nvx-rs: prepare the CLI's move onto the store and supervisor #675's bea29f13, with the artifacts of v0.1.0-dev.dad3e41d7f05:
  • WHP: the default test-microvm --backend whp set. managed-lifecycle failed twice in four runs because its guest console port was taken between the harness choosing it and OpenVMM binding it, which holding the port reproduces; the next runs passed.
  • KVM: every scenario of the default test-microvm --backend kvm set but cpu-profile-fallback. snapshot-core failed its clock check twice right after WSL restarted, while WSL stepped its clock, and passed once it settled.

Left to CI: the full matrix on KVM, MSHV, and WHP.

[Copilot]

🔵 Needs a closer look

The cross-platform lock and runtime-format switch affects VM recovery and termination and warrants final human review with full backend CI confirmation.

No open issues 🎉

What changed in this PR

Moves host::instance and its aci_edge_sandboxes consumer onto the shared store, supervisor, and session layers, continuing #620 after #675.

Changes:

• Unifies instance locking, launch recovery, reattachment, and shutdown; removes obsolete process helpers.
• Adds metadata-backed runtime identities and refuses incompatible runtime records.
• Updates documentation and tests for lifecycle recovery, identity mismatches, and stale-state cleanup.

File	Description
scripts/​test_nvx_tools.py	Updates harness expectations for artifact discovery and runtime records.
scripts/​nvx_tools/​aci_edge_sandboxes_tests.py	Passes the artifact directory and reads nested process IDs.
nvx-rs/​tests/​supervisor.rs	Removes checks for deleted process helpers.
nvx-rs/​tests/​store.rs	Uses supervisor spawning and verified process termination.
nvx-rs/​tests/​fake_openvmm.rs	Tests reattachment, identity mismatches, and forced shutdown.
nvx-rs/​src/​lib.rs	Updates the public lifecycle example.
nvx-rs/​src/​host/​store/​mod.rs	Updates imports and record-format documentation.
nvx-rs/​src/​host/​process.rs	Removes obsolete lifecycle helpers.
nvx-rs/​src/​host/​mod.rs	Documents shared instance supervision.
nvx-rs/​src/​host/​instance.rs	Delegates lifecycle operations to shared host layers.
nvx-rs/​src/​host/​artifacts.rs	Retains artifact origins for metadata-backed identities.
nvx-rs/​README.md	Documents the revised APIs and example.
aci_edge_sandboxes/​tests/​support/​mod.rs	Models interrupted launches using store journals.
aci_edge_sandboxes/​tests/​openvmm_unavailable.rs	Tests missing runtime identities for explicit paths.
aci_edge_sandboxes/​tests/​openvmm_fake.rs	Tests identity checks, recovery, and format rejection.
aci_edge_sandboxes/​tests/​openvmm_e2e.rs	Locates artifacts through their installation directory.
aci_edge_sandboxes/​src/​openvmm/​state.rs	Adopts store locks and stale-directory cleanup.
aci_edge_sandboxes/​src/​openvmm/​mod.rs	Routes backend operations through locked instances.
aci_edge_sandboxes/​src/​openvmm/​config.rs	Adds explicit or metadata-derived runtime identities.
aci_edge_sandboxes/​README.md	Documents identity requirements and lifecycle behavior.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-platform supervision, lock cleanup, and persisted-format switch warrant final human review with the full KVM, MSHV, and WHP CI results.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform lifecycle changes need human review and the pending CI matrix, and the forced-stop regression remains unresolved.

1 open issue

• Preserve verified VM when capability read fails during stop · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform process supervision, crash recovery, and incompatible persisted-state changes warrant final human review with full backend CI results.

No open issues 🎉

1 resolved since last review

• Preserve verified VM when capability read fails during stop

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The persisted-format switch and cross-platform locking, recovery, and termination changes need final human review with the full KVM/MSHV/WHP CI results.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The incompatible runtime-record switch and cross-platform supervision, recovery, and lock-removal changes warrant final human review and completion of the pending backend CI matrix.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform locking, crash recovery, forced termination, and the non-migrating record-format switch warrant human review and the pending full backend CI matrix.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform locking, crash recovery, and incompatible persisted-state changes warrant human sign-off and completion of the pending backend CI matrix.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform supervision, crash recovery, lock removal, and the persisted-format switch warrant human review with full backend CI confirmation.

No open issues 🎉

_{Review effort: Balanced}