[SECURITY] Protect admin and impersonation APIs and make proxy identity explicit #142

Summary

The cloud-init service currently exposes administrative and optional impersonation routes without application-level authentication or authorization.

Depending on deployment reachability, a caller can:

• Read cluster defaults, groups, and per-node instance data.
• Modify or delete configuration later consumed by nodes as root.
• Read sensitive group content, potentially including cluster credentials.
• Request another node's metadata through impersonation routes.
• Influence node identity by supplying forwarding headers when connecting directly to the service rather than through the packaged HAProxy.

Network isolation and HAProxy ACLs are useful defense in depth, but they should not be the authorization boundary for these APIs.

Priority: P0
Estimated complexity: XL

Prior Context

This is a successor to:

• [DEV]Protect insecure cloud-init endpoints for everything but GET #14 — protect insecure endpoints for non-GET operations
• [BUG] Enabling --jwks-url doesn't prevent requests to unsecure endpoints #49 — --jwks-url did not prevent access through unsecured routes

Those issues identified parts of the problem, but the current service still needs a complete route-authorization and trusted-proxy contract.

OpenCHAMI/release#80 sanitizes identity headers at the packaged HAProxy ingress. That protects proxied traffic, but direct backend requests can still supply X-Forwarded-For, Forwarded, or related identity headers.

This issue should follow the JWT conventions established by OpenCHAMI/roadmap#11 without prescribing which external identity provider issues the original credentials.

Required Security Contract

Node Bootstrap Routes

Node-facing cloud-init routes must remain usable before an operator JWT is available, but access must be bound to a trustworthy node identity.

Acceptable identity sources include:

• The transport peer address.
• A validated WireGuard peer.
• A short-lived, single-use bootstrap credential.
• A forwarding identity supplied by an explicitly trusted proxy.

An untrusted caller must not be able to select another node by supplying HTTP forwarding headers.

Administrative Routes

All /cloud-init/admin/** routes must require a valid JWT.

Authorization should distinguish at least:

• Read administrative configuration.
• Create/update/delete administrative configuration.
• Impersonate another node.

Proposed scope vocabulary, subject to the cross-project authorization ADR:

• cloud-init:read
• cloud-init:write
• cloud-init:impersonate

HTTP Semantics

• Missing, malformed, expired, incorrectly signed, or wrong-audience token: 401 Unauthorized
• Valid identity without the required scope: 403 Forbidden
• Error bodies must not expose token contents, claims, or internal policy.

Trusted Proxies

Add an explicit trusted-proxy configuration such as TRUSTED_PROXIES.

Requirements:

• Default is empty: forwarding headers are ignored.
• Forwarding headers are honored only when the immediate transport peer is within a configured trusted proxy CIDR.
• WireGuard/transport admission checks occur before any middleware rewrites RemoteAddr.
• Support IPv4 and IPv6.
• Invalid CIDRs fail startup.

Configuration

Security-sensitive configuration must fail closed.

• Enabling JWT authorization requires verifier/JWKS configuration.
• Unknown or ignored security keys must fail startup.
• Development-only insecure operation must require an explicit flag and emit a prominent warning.
• Production defaults must not silently expose administrative routes.

Acceptance Criteria

• Every route is classified as public health, node bootstrap, admin read, admin mutation, or impersonation.
• All admin routes require JWT authentication.
• Read, mutation, and impersonation scopes are independently enforced.
• Node bootstrap remains functional without an operator JWT.
• Node identity is derived from transport or another validated mechanism.
• Direct requests cannot change identity using forwarding headers.
• Trusted-proxy forwarding works only for configured proxy peers.
• WireGuard checks use the transport peer before RealIP processing.
• Missing/invalid credentials return 401.
• Insufficient scope returns 403.
• Authentication and authorization failures emit structured audit events without credentials or raw tokens.
• Route-inventory tests fail when a new route has no explicit policy.
• Tests cover proxied and direct requests, IPv4 and IPv6, spoofed headers, malformed tokens, wrong issuer/audience, and insufficient scopes.
• Documentation includes an operator migration guide.

Migration Requirements

Existing scripts using the admin API will need scoped tokens.

The migration documentation should include:

1. How to configure the verifier/JWKS source.
2. How automation obtains an appropriate token.
3. Which routes remain available to bootstrap nodes.
4. How to configure trusted HAProxy CIDRs.
5. How to confirm that direct backend access cannot spoof node identity.
6. A temporary containment procedure for deployments awaiting the secure release.

Non-Goals

• Selecting or operating a site's external identity provider.
• Treating HAProxy ACLs, source ports, or firewall rules as authentication.
• Defining authorization policy for other OpenCHAMI services.
• Solving release packaging or Quadlet configuration in this repository.

Security Impact

This should block a production security claim for affected releases. An unauthenticated caller with network reachability can potentially alter configuration executed as root during node boot.

Please coordinate disclosure and release timing with the release and OpenHPC recipe maintainers before publishing detailed exploit instructions.