Skip to content

feat(secrets): broker on by default, plus its VM test (#922) - #1041

Draft
lionello wants to merge 2 commits into
masterfrom
feat/secrets-default-on
Draft

lionello wants to merge 2 commits into
masterfrom
feat/secrets-default-on

Conversation

@lionello

Copy link
Copy Markdown
Collaborator

Part of #922. Draft until the new secrets VM check is green in CI.

What

  • secrets.enable now defaults to true on both backends (NixOS option and native Spec). secrets: {enable: false} turns it off.
  • New tests/secrets.nix (registered in vmTestsFor and the host lane): one node, two users (agent, robot). It checks socket/dir modes, root-only key, SO_PEERCRED refusing a second user and root, the systemd-run User= uid drop, scoped injection and redaction, control-group reaping on cancel and timeout, persistence across a broker restart, a corrupt record, and env-exec stripping (tombstones included).
  • Fixes found along the way:
    • On a box with web.enable off, /var/lib/agent-box-secrets was never created, so the broker failed 226/NAMESPACE in a restart loop (seen in a TCG boot). The tmpfiles rules are now secretsTmpfilesRules, applied by the broker block and added to internal.tmpfilesRules for the golden inventory.
    • SystemdLauncher ran python3 <worker> with a bare python3 that a NixOS service PATH lacks; it now execs the worker by absolute path.
  • Regenerated modules/agent-box.nix, tests/golden, tests/native; backend-parity allowlist updated (AGENT_BOX_SECRET_SOCKET stale entry removed, AGENT_BOX_SECRET_BROKER_ALLOW_DEV_LAUNCHER declared by design).

Security effect

Every box gets a root-owned broker per configured user by default. The root/uid boundary has never run end to end; this PR's VM test is the first attempt, so please do not merge before it is green.

Checks run (aarch64)

module-generated-up-to-date, golden-snapshot, backend-parity, one-spec-both-backends, secret-custody, assemble-module-escaping, module-single-file; python3 -m unittest tests.test_agentbox (189 pass); ruff --select F and ty on the extracted test script; nix eval .#checks.x86_64-linux.secrets.drvPath.

Not verified

The secrets VM test has not run to completion: it needs x86_64 KVM, and a TCG run on aarch64 does not boot in time. The SystemdLauncher change is untested until CI runs it. Comments saying the VM test is missing are left as they are until it passes.

🤖 Generated with Claude Code

lionello and others added 2 commits November 15, 2026 17:25
services.agent-box.secrets.enable and the native `secrets.enable` now
default to true; `secrets: {enable: false}` still turns it off.

- Move the two secret-custody tmpfiles rules into the module's named
  tmpfilesRules inventory (gated on secretsEnabled) so the golden
  snapshot and one-spec-both-backends see them; with the default off
  they were invisible to both.
- backend-parity: drop the stale AGENT_BOX_SECRET_SOCKET exception and
  declare AGENT_BOX_SECRET_BROKER_ALLOW_DEV_LAUNCHER (a dev gate the
  broker only reads, never supplied by any unit) as by-design.
- Regenerate modules/agent-box.nix, tests/golden and tests/native.

tests/secrets.nix (the VM test for the root/uid boundary) still does
not exist; this change does not add it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tests/secrets.nix boots one node with two Linux users and checks what
the unit tests (DirectLauncher, one uid) cannot: socket/dir modes, a
root-only key, SO_PEERCRED refusing root and a second user, the
systemd-run User= uid drop, scoped injection and redaction, control-group
reaping on cancel and timeout, store persistence across a broker restart,
a corrupt record, and env-exec stripping (tombstones included).
Registered in vmTestsFor and the `host` CI lane.

Found while running it under TCG on aarch64:
- A web-less box got no /var/lib/agent-box-secrets, so the broker failed
  226/NAMESPACE in a restart loop. The tmpfiles rules now live in
  secretsTmpfilesRules, applied by the broker block itself and added to
  internal.tmpfilesRules for the golden inventory only. (The previous
  commit on this branch had moved them into the web-gated list, which
  caused exactly this.)
- SystemdLauncher ran `python3 <worker>` with a bare python3 that a NixOS
  service PATH does not have; it now execs the worker by its absolute
  path (the generated script has a store-path shebang).

Not yet run to completion: the x86_64 test needs KVM, and under TCG the
guest boots too slowly to finish inside the driver's 3600s limit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@sonarqubecloud

Copy link
Copy Markdown

@defangdevs

Copy link
Copy Markdown
Owner

CI note on b0012f6 (triage session, nothing pushed): VM (host) fails in the new agent-box-secrets VM test, step "set / list / at-rest encryption / root key":

printf %s 's3cr3t-VALUE-0123456789' | agent-box-secret --profile t set K --stdin failed (2): agent-box-secret: secrets are not enabled on this box

So in the test VM the broker is still off, even though the PR turns it on by default. Likely the test machine config (or a module gate) does not take the new default. The other VM lanes' api.github.com resolve errors are the usual sandbox noise and are not the cause. VM (webhook) was still running when I looked.

@defangdevs

Copy link
Copy Markdown
Owner

CI run 43768360724 (VM host, agent-box-secrets) fails at test line 67:

printf %s 's3cr3t-VALUE-0123456789' | agent-box-secret --profile t set K --stdin failed (2): agent-box-secret: secrets are not enabled on this box

Cause (test bug, not the default flip): as_user() builds env + cmd as AGENT_BOX_SECRET_SOCKET=... printf ... | agent-box-secret .... A leading VAR=x applies only to the first command of a pipeline, so agent-box-secret has no socket variable and reports "not enabled". Fix: export it for the whole command line, e.g. env = f"export AGENT_BOX_SECRET_SOCKET=/run/agent-box-secrets/{user}.sock; ". Other non-piped calls are unaffected.

I did not push to your branch. Native checks passed; only VM (host) and the CI gate are red.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants