[Enrique Saurez (esaurez)]

Part of #620. Refs #535.

Every orchestrator of a sandbox keeps its configuration beside the sandbox store, and needs provisions, other transitions, and removals of the sandbox to keep the store's directory consistent, from any process. The CLI did that with a private lifecycle-lock and removal protocol around the store's lock. This PR moves that protocol into nvx::host::store, so the CLI, host::instance, aci_edge_sandboxes, and any other orchestrator share one implementation. It also makes the store own the file in which the VMM stages its outcome report, which deprovision used to refuse.

The store

• Caller files. CallerFiles declares the caller's configuration record, whose presence means that the sandbox is provisioned, and its other files in the directory: CallerFiles::none(), CallerFiles::new(config), .with(name). Each name is one plain file name in the directory, not . or .., of only ASCII letters and digits, ., _, and -, so that it names the same file on every host and no file system takes it for another name: file systems fold the case of other letters by rules of their own, macOS's take a composed letter for its decomposed form, and Windows also opens a file by its short name, such as RUNTIM~1.JSO for runtime.json. Ignoring ASCII case, it collides with none of the store's FILE_NAMES (the lock included), neither temporary form that the store owns, .nfs*, or another declared name. Declarations are checked when they are built, before any store is created, opened, locked, or removed with them.
• Locks. SandboxStore::lock_for_provision(dir, files, deadline) creates the directory if needed, and again if a concurrent removal deletes it, even while it is created and checked. SandboxStore::lock_existing(dir, files, deadline) fails with the new ErrorKind::NotFound when the directory is gone, also after it waited through a concurrent removal; the caller checks its configuration record under the lock. Both retry every 25 ms until the caller's deadline, including, on Windows file systems without POSIX deletion, the open of a removed lock file that other handles still hold. A lock file that another transition removes or replaces after an attempt opened it is locked again at once, and then every 25 ms until the deadline, as SandboxStore::lock now does too. A lock file that Windows refuses for good, such as a read-only one, which unlike a removed one opens without access rights, fails the lock with the denied access, also without a deadline.
• Release. A holder that leaves the directory without the configuration record removes its own lock file before it unlocks, unless it removed it already, so a never-provisioned or stale directory keeps no lock file.
• Removal. LockedStore::remove() needs the caller's configuration record: a lock taken without one, with CallerFiles::none() or the store's own SandboxStore::lock, refuses with RemovalError::Undeclared, since a transition that waits through the removal could not otherwise tell that the sandbox is gone. It refuses while a runtime record or a launch journal exists (RemovalError::Launched): callers reattach or recover first. It removes the store's files, the caller's files with the configuration record last, and the temporary files that the store owns. It then refuses foreign entries, naming them, and keeps the directory (RemovalError::Foreign), or removes its lock file while it still holds the lock, and then the directory. Lock files and .nfs* names are not foreign, since open files can keep the emptied directory for a while: NFS keeps a removed file that a handle still holds, such as a lock file, as .nfs*, Windows file systems without POSIX deletion keep a removed lock file under its name, and a transition that opens the lock file after it was removed creates a new one. The removal then releases the lock and returns a Lingering removal that holds the directory's identity. Lingering::wait(deadline) removes the directory once it empties, returns once the path names another directory, and reports RemovalError::ProvisionedAgain if a provision took the lock meanwhile, which it tells by looking the configuration record up by its path, so that a file system that ignores case finds it whatever the case of the name that the provision gave it. Each attempt to remove the directory first checks that the path still names the one that the removal holds, and a removal whose directory another removal deleted succeeds.
• The report staging file (its own commit). OpenVMM stages outcome.json in a tempfile named .tmp and six ASCII letters or digits, which a VMM killed while it writes the report leaves. The store owns names of exactly that form. It removes them once no process of the launch runs: when a recovery or an abandoned launch ends the launch (not when that is indeterminate), in clear_runtime, when the next launch begins, and at removal. It keeps them while a process of the launch may still run. Other .tmp* names stay foreign.
• SandboxStore::create and SandboxStore::open report a missing parent or directory as ErrorKind::NotFound rather than ErrorKind::Io, and so does the removal of staging files when the directory is gone. The store's format stays 4.

Callers

• CLI. LifecycleLock is a thin wrapper over the store's locks, with config.json as the configuration record. It keeps the Ctrl-C handling and the Python-compatible messages. deprovision finds the VM as before, then removes through LockedStore::remove. provision's already-provisioned check and the path checks of prepare_state_directory stay in the CLI. The goldens in nvx-rs/tests/fixtures/cli.json are unchanged.
• host::instance. Instance::create and Instance::open take the caller's CallerFiles, Instance::lock takes the lifecycle lock of an existing sandbox, and the new LockedInstance::remove(deadline) removes a stopped instance, with the caller's files and its directory, through the store. The new remove_interrupted_writes(path) removes the temporary files that interrupted writes of a caller's file with write_private or write_json left, exactly the names that write_private gives them, which the store refuses, as neither its own nor the caller's declared files.
• aci_edge_sandboxes. It declares sandbox.json as its configuration record, and replaces its own removal and stale-directory cleanup with the store's, after it removes the temporary files of interrupted writes of sandbox.json itself, under the lock.

Behavior differences

No golden or CLI message changes. In edge cases:

1. The release removes the lock file only while it is still the holder's own, by identity, and only when the configuration record is definitely missing.
2. On Windows file systems without POSIX deletion, a transition other than provision that meets a removed lock file now waits for it within its timeout, where it failed with access denied. A lock file that Windows refuses for good, such as a read-only one, still fails such a transition at once with access denied. A waiter for which Windows refuses to identify a removed lock file takes the lock again rather than failing.
3. A Unix fstat failure on the held state directory during deprovision now names the state directory.
4. The staging file: a refused start's leftover .tmpXXXXXX is removed once OpenVMM has exited, and at deprovision. The sandbox lifecycle acceptance now requires the refused managed start's outcome.json, rather than relying on deprovision to refuse the staging file. test_acceptance_detects_a_leaked_report_staging_file became the failed-start-kills-reporter case of test_acceptance_detects_lifecycle_regressions.
5. aci_edge_sandboxes:
   • a deprovision that finds files that are not the backend's now removes the sandbox's own files and sandbox.json first, as the CLI's does, so the ID is stale from then on and the directory keeps only those files;
   • deprovision removes the .*.tmp files that are the sandbox's, the store's and those of interrupted writes of sandbox.json, and now refuses other .*.tmp names, where it removed them all;
   • deprovision waits without a deadline for open files that keep the emptied directory, as its lock does;
   • the cleanup of a stale ID's directory also removes the store's leftovers, such as the temporary file of an interrupted replacement, and the one that a provision that died while it wrote sandbox.json left, and leaves alone a directory that is not private to the user.
6. A deprovision that finds a .nfs* entry, which NFS leaves for a removed file that a handle still holds, now waits for it within its timeout, where it refused it as a file not owned by NVX.
7. A deprovision whose emptied state directory another transition removed after deprovision removed its lock file now succeeds, where it failed with the operating system's not-found error.

API

• nvx::host::store::{CallerFiles, Lingering, RemovalError}
• SandboxStore::lock_for_provision, SandboxStore::lock_existing, LockedStore::files, LockedStore::remove, Lingering::{dir, wait}
• Instance::create(dir, name, files), Instance::open(dir, name, files), Instance::files, LockedInstance::remove, nvx::host::instance::remove_interrupted_writes
• ErrorKind::NotFound, Error::not_found

Validation

At this branch's head, on dev's 9b70ec97, the merge of #721:

• Windows 11 (x64, MSVC):
  • nvx-rs, with NVX_BUNDLE=skip: cargo fmt --all --check; cargo clippy --locked --all-targets --all-features -- -D warnings and cargo clippy --locked --lib --no-default-features -- -D warnings; cargo test --locked --all-features, then cargo test --locked, with the CLI goldens unchanged; cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings; cargo +1.89 check --locked --all-targets --all-features.
  • aci_edge_sandboxes: the same commands with --workspace, and python scripts/nvx.py test-agent-bundling --toolchain 1.95.0.
  • tests/runtime-adapter: the protocol-only and host-only configurations, as check-nvx-rs runs them.
  • Python: ruff check and ruff format --check of scripts .github/specula, pyright --pythonplatform Linux and --pythonplatform Windows, scripts/check_markdown_links.py, compileall, validate-nvx's unittest list, scripts/test_adversarial.py, test_hosts.py, and the nvx.py --help checks. All pass but test_setup_azure, which fails here with 18 failures, identically on dev without this PR.
• Linux, as a non-root user, on a Linux VM with the MSHV kernel: the same nvx-rs and aci_edge_sandboxes commands, plus check-nvx-rs's artifact-bundling test and both macOS cargo check --target aarch64-apple-darwin runs; all three runtime-adapter configurations, the guest one for x86_64-unknown-linux-musl; test-agent-bundling; compileall, and python3 -m unittest scripts/test_microvm_tests.py scripts/test_nvx_tools.py scripts/test_performance.py. Before the rebases onto nvx-rs: move host::instance onto the store and supervisor #721's later heads and dev, they also passed in WSL 2 (Ubuntu).
• Each commit builds and passes Clippy for nvx-rs and aci_edge_sandboxes on Windows.
• Real VMs, with the artifacts of v0.1.0-dev.d2fff11705bc, the newest release under dev, built from the same guest, kernel and OpenVMM inputs as 9b70ec97, whose OpenVMM provenance names the gitlink 7ff31ed7, and this branch's nvx-rs:
  • WHP (Windows 11): python scripts/nvx.py test-microvm --backend whp --scenario sandbox-lifecycle --scenario managed-exec-config --scenario managed-lifecycle, and python scripts/nvx.py test-aci-edge-sandboxes --backend whp (18 passed).
  • MSHV (a Linux VM with the MSHV kernel, as root, since /dev/mshv is root-only there): the same scenarios pass. python3 scripts/nvx.py test-aci-edge-sandboxes --backend mshv passes 13 of 18: as root, the five read-write mapping tests fail, since the workload cannot write to the directories that root creates on the host, exactly as at dev's 9b70ec97 without this PR, where the scenarios pass too. This VM cannot present its CPU's built-in profile (E_PROFILE_UNSUPPORTED), so OpenVMM ran with --cpu-profile host through a temporary wrapper.
  • KVM (WSL 2, nested), only before the rebases onto nvx-rs: move host::instance onto the store and supervisor #721's later heads and dev, with the artifacts of v0.1.0-dev.e714acbd47bf and the gitlink 9f04d05c: the same scenarios, and test-aci-edge-sandboxes --backend kvm (18 passed), with OpenVMM run through the same wrapper.

Left to CI: the KVM, MSHV and WHP matrix, the guest-binary steps, shellcheck, and the Specula wiring tests.

[Copilot]

🟡 Changes recommended

Unresolved directory-lifecycle races can prematurely fail provisioning or delete an empty replacement directory.

2 open issues

• Verify directory identity before removing replacements · New
• Retry creation NotFound when parent still exists · New

What changed in this PR

This PR centralizes sandbox lifecycle locking and removal in nvx::host::store, allowing the CLI, host::instance, and aci_edge_sandboxes to share the protocol described in #620.

Changes:

• Adds caller-file declarations, lifecycle locks, and removal with lingering-lock handling.
• Migrates callers to the shared store APIs and introduces ErrorKind::NotFound.
• Adds outcome-report staging cleanup, lifecycle regression coverage, and documentation.

File	Description
scripts/​test_microvm_tests.py	Updates staging-file cleanup and regression expectations.
scripts/​nvx_tools/​sandbox_lifecycle_tests.py	Requires an outcome report after a refused start.
nvx-rs/​tests/​fake_openvmm.rs	Supplies caller-file declarations to instance APIs.
nvx-rs/​src/​lib.rs	Updates the public instance example.
nvx-rs/​src/​host/​store/​tests.rs	Tests staging cleanup and missing-directory errors.
nvx-rs/​src/​host/​store/​mod.rs	Integrates lifecycle APIs and report staging cleanup.
nvx-rs/​src/​host/​store/​lifecycle/​tests.rs	Covers lifecycle locking, removal, and concurrent transitions.
nvx-rs/​src/​host/​store/​lifecycle.rs	Implements shared locks, release, and removal.
nvx-rs/​src/​host/​store/​caller.rs	Defines and validates caller-owned file names.
nvx-rs/​src/​host/​python/​fs.rs	Makes directory identity support available without CLI features.
nvx-rs/​src/​host/​instance.rs	Adopts lifecycle locks and exposes instance removal.
nvx-rs/​src/​host/​cli/​sandbox/​lock.rs	Wraps store lifecycle APIs while preserving CLI messages.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle.rs	Delegates deprovisioning to the store.
nvx-rs/​src/​error.rs	Adds the missing-state error kind and constructor.
nvx-rs/​README.md	Documents shared lifecycle APIs and caller responsibilities.
doc/​run.md	Explains lock release and outcome-report cleanup.
aci_edge_sandboxes/​src/​openvmm/​state.rs	Delegates removal and stale-directory cleanup to the store.
aci_edge_sandboxes/​README.md	Documents foreign-file handling during deprovisioning.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Filename alias validation and removal edge cases remain unresolved and can corrupt state or incorrectly fail deprovision.

3 open issues

• Reject Unicode aliases that bypass Windows filename collision checks · New
• Preserve pre-existing NFS lingering files during content removal · New
• Treat concurrent directory removal NotFound as successful · New

2 resolved since last review

• Verify directory identity before removing replacements
• Retry creation NotFound when parent still exists

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform locking and deletion require human review, particularly the unresolved Windows access-denied retry that can block indefinitely.

1 open issue

• Avoid retrying permanent Windows permission failures indefinitely · New

3 resolved since last review

• Reject Unicode aliases that bypass Windows filename collision checks
• Treat concurrent directory removal NotFound as successful
• Preserve pre-existing NFS lingering files during content removal

_{Review effort: Balanced}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

• SandboxStore::create/open were updated to map missing paths to ErrorKind::NotFound via… · New
• The new explicit error message for a missing outcome report is unlikely to be emitted as intended… · New
• In _FakeSandboxCli._deprovision, path.unlink() can raise if the .tmpXXXXXX entry disappears… · New

1 resolved since last review

• Avoid retrying permanent Windows permission failures indefinitely

_{Review effort: Lite}

[Copilot]

🟡 Changes recommended

Repeated lock-file replacements can bypass lifecycle deadlines and delay the CLI’s timeout and interruption handling.

1 open issue

• Enforce deadline when retrying replaced locks · New

3 resolved since last review

• The new explicit error message for a missing outcome report is unlikely to be emitted as intended…
• SandboxStore::create/open were updated to map missing paths to ErrorKind::NotFound via…
• In _FakeSandboxCli._deprovision, path.unlink() can raise if the .tmpXXXXXX entry disappears…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-process removal races and Windows/NFS file-handle semantics warrant final human review despite the extensive regression coverage.

No open issues 🎉

1 resolved since last review

• Enforce deadline when retrying replaced locks

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform locking and destructive cleanup warrant human review and the remaining platform CI matrix, with a reprovisioning classification issue still unresolved.

1 open issue

• Normalize config path comparisons for case-insensitive filesystems · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-process locking and Windows/NFS deletion semantics need final human review and platform CI confirmation, with an unresolved test race also requiring attention.

1 open issue

• Wait for released recovery before asserting staging file removal · New

1 resolved since last review

• Normalize config path comparisons for case-insensitive filesystems

_{Review effort: Balanced}

[Enrique Saurez (esaurez)]

Copilot's review of 785392d reported a previously missed issue in nvx-rs/src/host/store/lifecycle/tests.rs: where a removed file stays while a handle holds it, as on NFS or a Windows file system without POSIX deletion, the handle with which a provision waits for the removed lock file can keep the directory. Once the provision closes it, the lingering removal can remove the emptied directory and return Ok(()) before the provision creates it again. a_provision_that_waits_through_a_removal_provisions_the_sandbox_again and provisions_in_other_processes_that_race_a_removal_provision_the_sandbox_again accepted only ProvisionedAgain there.

Fixed in 780e20e: both accept Ok(()) | Err(RemovalError::ProvisionedAgain) from a lingering removal, and keep the checks of the provision's outcome and of the directory that it leaves. The other wait assertions in the file are deterministic: each provision, or foreign file, is in place before the file that stands in for an open .nfs one goes, and the waiters of the removal tests never provision.

[Copilot]

🔵 Needs a closer look

Cross-process locking and Windows/NFS deletion semantics warrant final human review and completion of the platform CI matrix at the final head.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-process locking and destructive removal across Windows and NFS semantics warrant final human review and confirmation of the remaining CI validation.

No open issues 🎉

_{Review effort: Balanced}