Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions aci_edge_sandboxes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -625,6 +625,7 @@ Defaults:
| `guest_network` | `10.0.0.2/24` |
| `start_timeout`, `control_timeout` | 60 s |
| `stop_timeout`, `exec_response_grace` | 30 s |
| `runtime_identity` | none: the install metadata of the artifacts' release or checkout names it; see [Runtime identity](#runtime-identity) |

Each timeout must be at most 30 days (`OpenVmmConfig::MAX_TIMEOUT`), and all
but `exec_response_grace` must be positive. `guest_network` must be an address
Expand All @@ -636,6 +637,36 @@ Choose a `state_root` that only the current user can access.
Windows and `$XDG_STATE_HOME/nvx/sandboxes` (default
`~/.local/state/nvx/sandboxes`) elsewhere.

### Runtime identity

Each sandbox's state directory is the `nvx::host::store` of its VM, which
records the runtime identity of the VM's start: the identities of its
OpenVMM, kernel, and initramfs, and the boot and control contract versions of
the build that started it. Every operation finds the sandbox's VM under the
backend's runtime identity, so a backend never drives a VM that another NVX
build, or other artifacts, started.

- Artifacts located in a release or a repository checkout, by
`OpenVmmConfig::discover` from `NVX_ARTIFACTS_DIR`, an `nvx/` directory, or
the bundle, by `from_release_dir` or `from_repo_layout`, or by
`from_artifacts` with artifacts that `nvx::Artifacts` located, take their
identity from the directory's install metadata: the digests that a
release's `SHA256SUMS` lists, or the build provenance that `nvx.py` records
beside a checkout's guest artifacts. Nothing reads or hashes an artifact.
Pointing `openvmm`, `kernel`, or `initrd` of such a configuration at other
files leaves it without an identity.
- Artifacts named by path, by `OpenVmmConfig::new` or by `discover` from
`NVX_OPENVMM`, `NVX_KERNEL`, and `NVX_INITRD`, have no install metadata. Set
`OpenVmmConfig::runtime_identity` to the identity that the deployment
records for them; it also replaces the identity of located artifacts.

Without a runtime identity, `probe`, `provision`, and `start` fail with
`backend_unavailable`, as does any operation on a sandbox whose store records a
VM. An operation on a sandbox whose VM runs under another runtime identity,
such as one that the build before an upgrade started, fails with
`backend_error`, naming the parts that differ and OpenVMM's process ID, and
leaves the VM running: stop it with the build and artifacts that started it.

### How it runs OpenVMM

- **Provision** validates the request and records it in
Expand All @@ -646,10 +677,12 @@ Windows and `$XDG_STATE_HOME/nvx/sandboxes` (default
system and starts the managed agent directly.
- A fresh 32-byte capability travels through OpenVMM's standard-input pipe,
which is closed before OpenVMM starts.
- Before launching OpenVMM, start records the control endpoint in
`launch.json`. If the caller dies before OpenVMM's process identity is
recorded, the next operation on the sandbox finds the VM through that
endpoint and terminates it, so no VM is left untracked.
- Before launching OpenVMM, start records the control endpoint and
OpenVMM's log in the store's launch journal, `launch.json`, and then
OpenVMM's process identity. If the caller dies before the VM is ready, the
next operation on the sandbox terminates the VM, through its process
identity, or through the endpoint's server if that process holds the log,
so no VM is left untracked.
- OpenVMM runs detached from the caller: in a new session on Linux, and
without the caller's console on Windows. It inherits no handle of the
caller except its standard input and log, so a caller whose own output is
Expand All @@ -672,9 +705,13 @@ Windows and `$XDG_STATE_HOME/nvx/sandboxes` (default
- **Deprovision** deletes the state directory. It refuses to remove a directory
that contains unexpected files.

The backend stores durable lifecycle state and reconciles it with the OpenVMM
process before every operation. Operations can therefore run safely from
different caller processes without forgetting a live VM. On Windows, set
Before every operation, the backend reattaches to the VM that the sandbox's
store records, as NVX's supervisor does after a restart: it checks that
OpenVMM's process still runs, verifying the process rather than trusting its
ID, under the backend's [runtime identity](#runtime-identity). A sandbox whose
VM died is provisioned again. Operations can therefore run safely from
different caller processes, one after another or after a caller restarts,
without forgetting a live VM. On Windows, set
`breakaway_from_job` if the caller runs inside a job object that would otherwise
terminate the VM when the caller exits.

Expand Down Expand Up @@ -950,8 +987,8 @@ egress default of `allow` and no deny rule that covers the guest network; see
| `start` or `deprovision` on a running sandbox | `AlreadyStarted` | `already_started` |
| `stop` on a stopped sandbox | `AlreadyStopped` | `already_stopped` |
| Unsupported policy or exec feature, oversized command, relative or oversized `process.cwd` | `PolicyValidation` | `policy_validation` |
| Missing OpenVMM artifacts, inaccessible hypervisor, incompatible release or guest image, unsupported host | `BackendUnavailable` | `backend_unavailable` |
| OpenVMM launch failure, boot timeout, control-session failure, I/O errors | `BackendError` | `backend_error` |
| Missing OpenVMM artifacts, inaccessible hypervisor, incompatible release or guest image, no [runtime identity](#runtime-identity), unsupported host | `BackendUnavailable` | `backend_unavailable` |
| OpenVMM launch failure, boot timeout, control-session failure, a VM of another runtime identity, I/O errors | `BackendError` | `backend_error` |

Start failures name the OpenVMM log, `OpenVmmBackend::log_path`.

Expand Down
65 changes: 57 additions & 8 deletions aci_edge_sandboxes/src/openvmm/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ use std::time::Duration;
use aci_edge_sandboxes_model::spec::{is_valid_guest_network, is_valid_hostname};
use nvx::Hypervisor;
use nvx::host::artifacts::{Artifacts, absolute};
use nvx::host::instance::SOCKET_NAME;
use nvx::host::launch::{self, WorkloadIdentity};
use nvx::host::store::{RuntimeIdentity, SOCKET_NAME};

use crate::error::{Error, Result};

Expand All @@ -14,7 +14,8 @@ use crate::error::{Error, Result};
/// Construct it with [`OpenVmmConfig::discover`], [`OpenVmmConfig::from_artifacts`],
/// [`OpenVmmConfig::new`], [`OpenVmmConfig::from_release_dir`], or
/// [`OpenVmmConfig::from_repo_layout`], then adjust the public fields. Defaults match
/// `scripts/nvx.py sandbox`.
/// `scripts/nvx.py sandbox`. Artifacts named by path, as by [`OpenVmmConfig::new`], also need
/// their [`runtime_identity`](Self::runtime_identity).
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub struct OpenVmmConfig {
Expand All @@ -24,6 +25,19 @@ pub struct OpenVmmConfig {
pub kernel: PathBuf,
/// The NVX Alpine initramfs (`initramfs.cpio.gz`): the guest userland and its control agent.
pub initrd: PathBuf,
/// The runtime identity of the artifacts, which the store records with each sandbox's VM:
/// the identities of OpenVMM, the kernel, and the initramfs, with the contract versions of
/// this build. A VM that runs under another runtime identity, as after an upgrade, is reported
/// and neither driven nor stopped.
///
/// Leave it unset for artifacts located in an NVX release or repository checkout, as by
/// [`from_release_dir`](Self::from_release_dir), [`from_repo_layout`](Self::from_repo_layout),
/// [`discover`](Self::discover), or [`from_artifacts`](Self::from_artifacts): every operation
/// that needs it then reads it from the install metadata of the release or checkout, a
/// release's `SHA256SUMS` or a checkout's build provenance, without hashing an artifact, so an
/// install upgraded in place is identified anew. Artifacts named by path have no install
/// metadata: without it, the backend is unavailable.
pub runtime_identity: Option<RuntimeIdentity>,
/// Hypervisor used by OpenVMM.
pub hypervisor: Hypervisor,
/// Directory that holds per-sandbox state. Choose a directory private to the current user.
Expand Down Expand Up @@ -72,6 +86,9 @@ pub struct OpenVmmConfig {
/// hosts without a hypervisor. Available only with the `testing` feature.
#[cfg(feature = "testing")]
pub skip_hypervisor_probe: bool,
/// The artifacts as [`from_artifacts`](Self::from_artifacts) received them, with the release
/// or checkout in which they were located, whose install metadata identifies them.
located: Option<Artifacts>,
}

impl OpenVmmConfig {
Expand Down Expand Up @@ -103,6 +120,7 @@ impl OpenVmmConfig {
openvmm: openvmm.into(),
kernel: kernel.into(),
initrd: initrd.into(),
runtime_identity: None,
hypervisor,
state_root: state_root.into(),
memory_mib: Self::DEFAULT_MEMORY_MIB,
Expand All @@ -119,22 +137,53 @@ impl OpenVmmConfig {
breakaway_from_job: false,
#[cfg(feature = "testing")]
skip_hypervisor_probe: false,
located: None,
}
}

/// Creates a configuration from located [`Artifacts`].
/// Creates a configuration from [`Artifacts`], whose release or checkout, if they were located
/// in one, identifies them; see [`runtime_identity`](Self::runtime_identity).
pub fn from_artifacts(
artifacts: Artifacts,
hypervisor: Hypervisor,
state_root: impl Into<PathBuf>,
) -> Self {
Self::new(
artifacts.openvmm,
artifacts.kernel,
artifacts.initrd,
let mut config = Self::new(
artifacts.openvmm.clone(),
artifacts.kernel.clone(),
artifacts.initrd.clone(),
hypervisor,
state_root,
)
);
config.located = Some(artifacts);
config
}

/// Returns the runtime identity of the configured artifacts:
/// [`runtime_identity`](Self::runtime_identity) if it is set, and otherwise the one that the
/// install metadata of the release or checkout in which they were located names, read now.
///
/// Fails with [`nvx::ErrorKind::Unavailable`] for artifacts named by path without one, and
/// when the metadata is missing or unreadable, or does not name the configured paths.
pub(crate) fn identify(&self) -> nvx::Result<RuntimeIdentity> {
if let Some(identity) = &self.runtime_identity {
return Ok(identity.clone());
}
let Some(mut artifacts) = self
.located
.clone()
.filter(|artifacts| artifacts.artifact_dir().is_some())
else {
return Err(nvx::Error::unavailable(
"the artifacts are named by path, so no install metadata identifies them: set \
OpenVmmConfig::runtime_identity, or locate them in an NVX release or repository \
checkout",
));
};
artifacts.openvmm.clone_from(&self.openvmm);
artifacts.kernel.clone_from(&self.kernel);
artifacts.initrd.clone_from(&self.initrd);
artifacts.runtime_identity()
}

/// Creates a configuration with no NVX-specific input: the artifacts come from
Expand Down
Loading
Loading