Skip to content

nvme: decode the device data in a sandbox - #4183

Merged
igaw merged 8 commits into
linux-nvme:masterfrom
dwsuse:privsep-sandbox
Oct 21, 2026
Merged

igaw merged 8 commits into
linux-nvme:masterfrom
dwsuse:privsep-sandbox

Conversation

@dwsuse

@dwsuse dwsuse commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fifth step of privilege separation (#3879), re-scoped: the printers which decode device data run in a sandboxed child process. The previous version sandboxed the parsers of the command line, nvme-cli.conf and --set-options; that is dropped. Under the threat model (see Documentation/PRIVSEP.md) those inputs come from the user who runs nvme-cli or from a file owned by root, so sandboxing them gained nothing, while it needed the most intricate code (copying the regions back, the result check). The data a device or a fabrics target returns is what an attacker controls, and most of the bugs fuzzing found were in its decoders.

Behind the sandbox meson option (disabled by default for now, Linux and FreeBSD). NVME_SANDBOX=0 disables the sandbox, e.g. for a debugger; NVME_SANDBOX=1 enables it in any build.

CI: as privilege separation and the sandbox are disabled by default, no job built them. The last commit adds scripts/build.sh privsep, which enables both, and the Build privsep workflow: on Linux it runs the tests as a user (the privsep tests skip as root), on FreeBSD it builds the sandbox only, and like build-freebsd.yml only when master is updated.

The primitive: shr_sandbox_begin() forks a child which closes all descriptors but stdio and the result pipe, enters the sandbox (seccomp on Linux, Capsicum on FreeBSD) and runs the code that follows; shr_sandbox_end() hands one result back. The child reads the device data in its copy of the memory and prints itself. Without a sandbox (disabled, unsupported, valgrind, Windows) the code runs in process. A crash, a missing or malformed result is an error.

The printers: the hook is the nvme_print() macro, which all nvme_show_*() functions use, including the log plugin. Printers called from a sandboxed printer run in the same child. The libnvme tree printers (they read sysfs while printing), the ZNS zone list printers (they hand JSON objects to their caller), top and the messages stay in process, unless a sandboxed printer prints them. A printer which fails in the sandbox is reported like any other error (nvme_show_error()) and nvme-cli exits with an error.

JSON output: a command prints one JSON document, and the format stays as it is. In process nothing changes: the printers add their members to the document. A sandboxed child only has a copy of the document, so struct print_ops gets optional capture hooks, which the dispatcher calls around a sandboxed printer and whose result it hands over as it is. Only the JSON plugin implements them, and its details stay in nvme-print-json.c: in the child the printer call writes to an object of its own, whose members are serialized; the caller adds them to its document as raw text, re-indented to their level. No parse in the caller: a parse would turn 128 bit numbers into doubles, and the child can write anything to stdout anyway. The stdout and binary plugins need no hooks, the child writes to the inherited stdout. Two fixes found along the way were merged separately in #4231.

Closed standard descriptors: a new first commit opens /dev/null on the standard descriptors nvme-cli starts with closed. Before, the socket to the privsep helper took the place of a closed one: with stdout closed the output went into the requests to the helper, and the sandboxed child, which keeps stdio, would have kept the socket. The child closes the other descriptors with close_range(), else those listed in /proc/self/fd or with closefrom(), and fails if it cannot.

What it buys: with plain root the decoders no longer run as root; with privilege separation a decoder that is taken over no longer holds the helper socket, which issues any NVMe command; a crashing decoder no longer takes nvme-cli down. Not covered: size handling in the command code that fetches the data, vendor plugins which decode inside the command, the printers which stay in process.

The comparison test below found a bug on master (the JSON output of the persistent event log printed uninitialized memory), fixed in #4229.

Testing:

  • sandbox unit test: results, the time zone of the caller (libc loads it on first use, so the caller loads it before the fork), fallback, nesting, empty and oversized results, data after the result, crashes, denied open, closed descriptors, terminal queries on a pty, other ioctls, tgkill of another process (the result framing checks mutation-checked)
  • in process the output is unchanged; the print test checks that the JSON output is indented by its depth (a broken re-indent fails it), and dropping the members a sandboxed printer hands back fails print-sandbox, print-sandbox-compare, host-metadata-sandbox and discovery-logs-sandbox (mutation-checked)
  • closed stdio: test_closed_stdio closes fd 0, 1 and 2 in turn and checks from the helper that the descriptor is /dev/null, not the helper socket (fails on master); a descriptor above a lowered soft limit is closed in the child, also by the fallback without close_range() (sandbox-util-fallback, fails with the old loop); the sandbox unit test skips only when no sandbox is available, other failures fail it
  • print-sandbox-compare: the print test (about 1100 invocations on random device data) with and without the sandbox; exit status, stdout and stderr must be the same
  • every mock CLI test also runs with NVME_SANDBOX=1 (<name>-sandbox), and the print test checks that the sandbox is used (it fails when nvme-cli cannot fork)
  • an attack planted in the smart-log printer (open a file, write to every inherited descriptor, crash): without the sandbox it reads the file, can write to the helper socket and kills nvme-cli; with it, open fails with EPERM, the helper socket is gone, the crash is contained
  • full test suite with the sandbox enabled and disabled (gcc, --werror), without json-c, every commit on its own; a decoder crash forced in the child prints the labelled error in text mode and an "error" member in JSON mode
  • previous revision, the sandbox primitive is unchanged since (CI covers the cross builds of this one): clang and privsep builds; FreeBSD: tests, and on a real device id-ctrl, smart-log, error-log, fw-log, id-ns, get-feature print the same with and without the sandbox; truss shows the child entering cap_enter()
  • previous revision: cross builds and tests for s390x, ppc64le, armhf, i386 (under qemu-user there is no seccomp, so the sandbox falls back to in-process there), Windows (x86_64, aarch64) builds
  • cost: about 0.8 ms per command

Comment thread shared/sandbox-util.c Outdated
Comment thread src/sandbox.h Outdated
Comment thread src/argconfig.c
Comment thread src/global-config.c Outdated
@igaw
igaw requested a balanced review from Copilot October 15, 2026 07:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved confinement gaps, fail-open setup handling and lost JSON diagnostics prevent approval.

5 open findings
What changed in this PR

Adds optional parser sandboxing as the next privilege-separation step, building on #4182.

Changes:

  • Runs argument, configuration and --set-options parsers in confined children with parent-side report validation.
  • Adds Linux seccomp, FreeBSD Capsicum and runtime sandbox controls.
  • Expands unit and CLI tests for sandbox behavior and parsing equivalence.
File Description
tests/​unit/​c/​test-plugin.c Tests command lookup and sandboxed dispatch.
tests/​unit/​c/​test-global-config.c Runs configuration tests in both modes.
tests/​unit/​c/​test-argconfig-report.c Tests rejection of malformed reports.
tests/​unit/​c/​test-argconfig-parse.c Compares sandboxed and direct parsing.
tests/​unit/​c/​meson.build Registers parser tests and dependencies.
tests/​cli/​nvme_sandbox_test.py Tests output equivalence and sandbox use.
tests/​cli/​nvme_privsep_test.py Tests privileged-helper integration.
tests/​cli/​meson.build Adds sandbox and helper test variants.
src/​sandbox.h Declares CLI sandbox controls.
src/​sandbox.c Implements sandbox enablement policy.
src/​privsep.h Declares privilege-separation interfaces.
src/​privsep.c Sandboxes privilege-drop decisions.
src/​plugin.h Adds command privilege metadata.
src/​plugin.c Implements command lookup and privilege checks.
src/​nvme.c Initializes privilege separation early.
src/​nvme-cmds-fabrics.c Marks commands requiring retained privileges.
src/​nvme-cmds-enumerate.c Marks enumeration privilege requirements.
src/​nvme-cmds-deprecated.c Marks deprecated key-command privilege requirements.
src/​meson.build Includes sandbox and privilege-separation sources.
src/​global-ctx.c Sandboxes --set-options parsing.
src/​global-config.c Sandboxes configuration parsing and validates reports.
src/​argconfig.c Serializes and validates sandboxed parser results.
shared/​tests/​test-sandbox-util.c Tests confinement and result handling.
shared/​tests/​test-ini-util.c Tests separate configuration-file reading.
shared/​tests/​meson.build Registers sandbox utility tests.
shared/​sandbox-util.h Defines the shared sandbox interface.
shared/​sandbox-util.c Implements confinement and child-result transport.
shared/​sandbox-util-win.c Provides Windows unsupported fallback.
shared/​meson.build Selects platform sandbox implementation.
shared/​ini-util.h Declares separate INI-file reading.
shared/​ini-util.c Separates file reading from parsing.
plugins/​zns/​zns.c Marks listing privilege requirements.
plugins/​wdc/​wdc-nvme.c Marks telemetry privilege requirements.
plugins/​utils/​utils.c Marks metadata-command privilege policy.
plugins/​solidigm/​solidigm-nvme.c Marks version-command privilege policy.
plugins/​seagate/​seagate-nvme.c Marks version-command privilege policy.
plugins/​sandisk/​sandisk-nvme.c Marks telemetry privilege requirements.
plugins/​registry/​registry-nvme.c Marks registry-list privilege requirements.
plugins/​netapp/​netapp-nvme.c Marks enumeration privilege requirements.
plugins/​nbft/​nbft-plugin.c Marks NBFT privilege requirements.
plugins/​micron/​micron-nvme.c Marks privileged and no-device commands.
plugins/​keys/​keys-plugin.c Marks key-management privilege requirements.
plugins/​huawei/​huawei-nvme.c Marks listing privilege requirements.
plugins/​exclusion/​exclusion-nvme.c Marks exclusion-management privilege requirements.
plugins/​config/​config-nvme.c Marks configuration-command privilege requirements.
meson.build Configures sandbox support.
meson_options.txt Adds the disabled-by-default sandbox option.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread shared/sandbox-util.c Outdated
Comment thread shared/sandbox-util.c Outdated
Comment thread shared/sandbox-util.c Outdated
Comment thread shared/sandbox-util.c Outdated
Comment thread src/global-ctx.c Outdated
@dwsuse
dwsuse force-pushed the privsep-sandbox branch 3 times, most recently from eef62ea to 29f62e2 Compare October 20, 2026 11:48
@dwsuse
dwsuse marked this pull request as ready for review October 20, 2026 11:51
@dwsuse dwsuse changed the title nvme: parse untrusted input in a sandbox nvme: decode the device data in a sandbox Oct 20, 2026
@dwsuse
dwsuse force-pushed the privsep-sandbox branch 3 times, most recently from 3c9ade1 to ec05e84 Compare October 20, 2026 16:49
@dwsuse
dwsuse force-pushed the privsep-sandbox branch 2 times, most recently from bd513d9 to ac500e5 Compare October 21, 2026 11:44
@dwsuse
dwsuse requested a balanced review from Copilot October 21, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread shared/sandbox-util.c
Comment thread shared/sandbox-util.c Outdated
Comment thread shared/tests/test-sandbox-util.c Outdated
A descriptor which nvme-cli opens takes the lowest free number. If nvme-cli
starts with a standard descriptor closed, the socket to the privsep helper
becomes that descriptor: with stdout closed, the output, the strings of
the device included, is written into the requests to the helper.

Open /dev/null on the closed standard descriptors before anything else.
The test closes each in turn and checks that the descriptor is not the
socket to the helper.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
shr_read_all() reads text: it NUL-terminates and discards what does not
fit. Add shr_read_full(), which reads binary data up to a length or to
the end of the file and returns how much it read.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
Add shr_sandbox_begin() and shr_sandbox_end(), which run the code
between them in a forked child process that can only allocate memory,
write to stdout and stderr and exit. The child sees the memory of the
caller as it was at the fork, so a decoder reads the data a device
returned without copying it, and writes its output itself. It can hand
one result back to the caller with shr_sandbox_end(). Without a sandbox,
i.e. when disabled, on systems without support or under valgrind, the
code runs in the caller. The caller loads the time zone before the
fork, which the child could not read.

The child keeps only stdio and the result pipe open; in particular it
cannot use the privsep helper socket. It closes the other descriptors
with close_range(), else those listed in /proc/self/fd or with
closefrom(), and fails if it cannot: a loop up to the limit would miss
descriptors above a lowered limit. The caller keeps stdio open, else a
descriptor opened in its place would stay open in the child. On Linux it
installs a seccomp filter: memory management, write, the terminal
queries of stdio and word wrapping (TCGETS, TCGETS2, TIOCGWINSZ),
getrandom, signals to itself for abort(), what sanitizer runtimes need,
and exit. Other system calls fail with EPERM, so libc falls back
gracefully, e.g. gettext(). A filter rather than strict mode, which
cannot be entered under an existing filter (systemd, containers). On
FreeBSD the child limits the rights of its descriptors and enters
Capsicum capability mode. Windows has no fork(), so the code runs in
process there.

Only a child which did not enter the sandbox may report that it found
none; any other failure of the child, a crash, an exit without a result
or a malformed result, is an error.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
Add the sandbox meson option (Linux and FreeBSD, disabled for now) and
enable the sandbox in main() accordingly. NVME_SANDBOX=0 disables it,
e.g. for a debugger; NVME_SANDBOX=1 enables it in any build. Nothing
uses the sandbox yet.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
The printers decode what a device returned: identify data, log pages,
features. A device, or a fabrics target, controls that data, and most of
the bugs fuzzing found were in these decoders. They run as root unless
privilege separation dropped the privileges, and even then they hold
the socket to the privsep helper, which issues any NVMe command.

Run the printers in the sandbox, if enabled: nvme_print() forks a child
which enters the sandbox and runs the printer. The child cannot open
files or use the helper socket, and a printer which crashes takes only
the child down. nvme-cli then reports the failure and exits with an
error.

What the child writes to stdout appears as it does in process. Output
which the printer collects instead, the JSON document, the child cannot
hand back by itself. Add optional capture hooks to struct print_ops: the
child captures the output of the printer call, and the caller adds it to
its own. The JSON printer gives the printer call an object of its own in
the child, serializes its members and splices them into the document as
text, not parsed again, so 128 bit numbers stay exact. As the members are
put together as text, the print test now also checks that the JSON output
is indented by its depth.

Some printers stay in process: the ones of the libnvme tree, which read
sysfs while printing and decode no device data, the ZNS zone list ones,
which hand JSON objects to their caller, the interactive top, the
document and the messages.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
Run the mock CLI tests a second time with NVME_SANDBOX=1, and compare
the output of the print test, which runs the decoders on random device
data, with and without the sandbox: the exit status, stdout and stderr
must be the same. The list commands are left out of the comparison, as
they generate a missing host NQN anew for every run.

Also check that the sandbox is used: when nvme-cli cannot fork, the
decoders fail with the sandbox and work without it.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
Add Documentation/PRIVSEP.md for reviewers: the threat model, the
privsep helper, its wire protocol and the lifecycle in nvme-cli, and how
the decoder sandbox works: the child and what it hands back, which
printers run in it, how the JSON output of a sandboxed printer gets
into the document, and what the sandbox allows.

Point to it from the README, from SECURITY.md for its threat model, and
from the code of the sandbox.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
Privilege separation and the decoder sandbox are disabled by default, so
no CI job builds them, and the privsep tests do not run. Add the privsep
config to scripts/build.sh, which enables both, privilege separation on
Linux only, and a workflow which runs it on Linux and FreeBSD.

The privsep tests skip as root, as nvme-cli ignores the test helper then:
the config refuses to run as root on Linux, and the Linux job runs it as
a user. It runs without valgrind, under which the sandbox runs in
process. The FreeBSD job builds the sandbox only and runs, as
build-freebsd.yml, only when master is updated.

Signed-off-by: Daniel Wagner <dwagner@suse.de>
@igaw
igaw merged commit 90a4ab1 into linux-nvme:master Oct 21, 2026
41 checks passed
@dwsuse
dwsuse deleted the privsep-sandbox branch October 23, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants