diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 0b91ed1390..61bcdf8e60 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -29,6 +29,12 @@ every setting with its environment variable. | `--ingress-allowed-spiffe-ids IDS` | Comma-separated SPIFFE IDs (exact match) of the clients the ingress router lets reach actors; any listed client may reach any actor. Unset, no client may reach actors through the router. Both dataplanes require a Pod Identity CA client certificate on HTTPS and TLS CONNECT. The SPIFFE URI SAN allowlist may be checked during or after TLS, including after CONNECT acceptance, but unlisted clients must be denied before actor routing. The podidentity signer issues `spiffe://cluster.local/ns//sa/`. `ATE_INGRESS_ALLOWED_SPIFFE_IDS` when the flag is absent | | `--egress-gateway-extra-spiffe-ids IDS` | Comma-separated SPIFFE IDs that hold ate-api-server's `egress_gateway` role next to atenet-egress, such as a credential provider that mints actor JWTs. ate-setup passes atenet-egress's ID and then each listed ID as repeated `--egress-gateway-spiffe-id` flags; unset passes none, leaving ate-api-server's default. `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` when the flag is absent | | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter | +| `--experimental-additional-egress-extproc-allow-mode-override` | Let that processor buffer the request and response bodies of a request it picks, through a `mode_override` in its `request_headers` response. See [`docs/security/extensibility.md`](../../docs/security/extensibility.md). Requires `--experimental-additional-egress-extproc-service`. `ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE=true` when the flag is absent | +| `--experimental-additional-egress-extproc-timeout DURATION` | How long Envoy waits for each response from that processor (default `2s`, at most `1h`). Requires `--experimental-additional-egress-extproc-service`. `ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT` when the flag is absent | +| `--openappa` | Deploy [OpenAPPA](../../docs/guides/openappa.md) as the egress gateway's additional ext_proc filter, on `deploy ate-system` and `deploy atenet`. It sets the three `--experimental-additional-egress-extproc-*` flags above, refusing another service or a mode override turned off, and a redeploy without it removes OpenAPPA. Requires `--atenet-dataplane=envoy`. `ATE_OPENAPPA=true` when the flag is absent | +| `--openappa-policy PATH` | appa.toml to apply on every deploy. Unset, the bundled policy is applied only when the cluster has none. `ATE_OPENAPPA_POLICY` when the flag is absent | +| `--openappa-mcp-servers HOST=NAME,...` | MCP servers OpenAPPA checks tool calls to (default `api.githubcopilot.com=github`). `ATE_OPENAPPA_MCP_SERVERS` when the flag is absent | +| `--openappa-runtime-image IMAGE`, `--openappa-extproc-image IMAGE` | Override OpenAPPA's images. `ATE_OPENAPPA_RUNTIME_IMAGE` and `ATE_OPENAPPA_EXTPROC_IMAGE` when the flags are absent | | `--credential-provider JSON` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. A JSON array of provider objects selects several, each serving the `ate-secret://` URIs its name is the host of. See [`docs/guides/egress-credential-injection.md`](../../docs/guides/egress-credential-injection.md). `ATE_CREDENTIAL_PROVIDER` when the flag is absent | | `--ateapi-log-level LEVEL` | ate-api-server log level: `debug`, `info`, `warn`, or `error`. Written to the `ate-api-config` ConfigMap on every deploy when set; unset keeps the installed level (`info` on a new install). Changing the level updates the env-hash annotation and rolls `ate-api-server`. `ATE_API_LOG_LEVEL` when the flag is absent | | `--otlp-endpoint URL` | Send control plane telemetry to `URL` instead of the cluster default (see [`benchmarking/telemetry/README.md`](../../benchmarking/telemetry/README.md)). `ATE_OTLP_ENDPOINT` when the flag is absent | diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index 30bcd18aad..7840d18966 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -238,6 +238,25 @@ type Config struct { // AdditionalEgressExtprocService is the optional NS/SVC:PORT external processor filter. AdditionalEgressExtprocService string + // AdditionalEgressExtprocAllowModeOverride lets that processor switch a + // request to buffered body processing with a mode_override. + AdditionalEgressExtprocAllowModeOverride bool + + // AdditionalEgressExtprocTimeout is that filter's message_timeout and gRPC + // timeout. + AdditionalEgressExtprocTimeout time.Duration + + // OpenAPPA deploys OpenAPPA as the additional ext_proc filter + // (ATE_OPENAPPA). It implies the three settings above. + OpenAPPA bool + // OpenAPPAPolicy is a local appa.toml. Empty means the bundled policy, + // applied only where the cluster has none. + OpenAPPAPolicy string + // OpenAPPAMCPServers are the HOST=NAME pairs OpenAPPA checks tool calls to. + OpenAPPAMCPServers []string + OpenAPPARuntimeImage string + OpenAPPAExtprocImage string + // CredentialProviderJSON is the --credential-provider value; see // CredentialProviders for its schema. CredentialProviderJSON string @@ -532,6 +551,15 @@ func (c *Config) ScriptEnv() []string { } if c.AdditionalEgressExtprocService != "" { merged["ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"] = c.AdditionalEgressExtprocService + merged["ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT"] = c.AdditionalEgressExtprocTimeout.String() + } + delete(merged, "ATE_OPENAPPA") + if c.OpenAPPA { + merged["ATE_OPENAPPA"] = "true" + } + delete(merged, "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE") + if c.AdditionalEgressExtprocAllowModeOverride { + merged["ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE"] = "true" } if c.CredentialProviderJSON != "" { merged["ATE_CREDENTIAL_PROVIDER"] = c.CredentialProviderJSON @@ -576,6 +604,18 @@ const ( K8sCredentialProviderAddress = "k8s-credential-provider." + installdefaults.SystemNamespace + ".svc:50051" ) +// OpenAPPA's Service in manifests/openappa, which the egress gateway reaches +// as its additional ext_proc filter, and its default images. +const ( + OpenAPPAExtprocService = installdefaults.SystemNamespace + "/openappa:9443" + OpenAPPARuntimeImage = "ghcr.io/archestra-ai/appa-runtime:v0.42.0@sha256:4cc8f3f685a725106aba5ea05f022447799d11a70a34a0123c96bd73ff27b5ca" + OpenAPPAExtprocImage = "ghcr.io/archestra-ai/appa-extproc:v0.43.0" + + // openAPPAExtprocTimeout leaves room for the policy's external calls, + // such as the GitHub battery's. + openAPPAExtprocTimeout = "60s" +) + // credentialProviderKey is the setting the provider JSON arrives on. const credentialProviderKey = "atenet.egress.credentialProvider" diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index 18e95e41be..ab52544d8d 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -46,7 +46,9 @@ func loadEnv(t *testing.T) { "ACTOR_JWT_ALGORITHM", "ACTOR_JWT_ISSUER", "ANTHROPIC_API_KEY", + "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE", + "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT", "ATE_API_AUTHZ_BOOTSTRAP_OWNERS", "ATE_API_LOG_LEVEL", "ATE_API_POSTGRES_CLOUDSQL_GSA", @@ -72,6 +74,11 @@ func loadEnv(t *testing.T) { "ATE_INSTALL_KIND", "ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER", "ATE_INSTALL_ROLLOUT_TIMEOUT", + "ATE_OPENAPPA", + "ATE_OPENAPPA_EXTPROC_IMAGE", + "ATE_OPENAPPA_MCP_SERVERS", + "ATE_OPENAPPA_POLICY", + "ATE_OPENAPPA_RUNTIME_IMAGE", "ATE_OTLP_ENDPOINT", "ATE_PREVIEW", "BENCHMARK_ACTOR_MEMORY", @@ -278,6 +285,101 @@ func TestLoadE2ETuning(t *testing.T) { } } +func TestLoadAdditionalEgressExtproc(t *testing.T) { + const service = "ate-system/extproc:50051" + for _, tc := range []struct { + name string + opts Options + env map[string]string + wantOverride bool + wantTimeout time.Duration + }{ + {name: "defaults", opts: Options{AdditionalEgressExtprocService: service}, wantTimeout: 2 * time.Second}, + { + name: "flags", + opts: Options{ + AdditionalEgressExtprocService: service, + AdditionalEgressExtprocAllowModeOverride: true, + AdditionalEgressExtprocTimeout: "10s", + }, + wantOverride: true, wantTimeout: 10 * time.Second, + }, + { + name: "environment", + opts: Options{AdditionalEgressExtprocService: service}, + env: map[string]string{ + "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "true", + "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "500ms", + }, + wantOverride: true, wantTimeout: 500 * time.Millisecond, + }, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + for k, v := range tc.env { + t.Setenv(k, v) + } + cfg, err := Load(tc.opts) + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if cfg.AdditionalEgressExtprocAllowModeOverride != tc.wantOverride { + t.Errorf("AdditionalEgressExtprocAllowModeOverride = %v, want %v", cfg.AdditionalEgressExtprocAllowModeOverride, tc.wantOverride) + } + if cfg.AdditionalEgressExtprocTimeout != tc.wantTimeout { + t.Errorf("AdditionalEgressExtprocTimeout = %v, want %v", cfg.AdditionalEgressExtprocTimeout, tc.wantTimeout) + } + if _, exported := scriptEnvMap(t, cfg)["ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE"]; exported != tc.wantOverride { + t.Errorf("ScriptEnv() exports ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE = %v, want %v", exported, tc.wantOverride) + } + }) + } +} + +// --openappa points the additional ext_proc filter at OpenAPPA, keeping any +// timeout a channel supplied. +func TestLoadOpenAPPA(t *testing.T) { + for _, tc := range []struct { + name string + opts Options + wantTimeout time.Duration + }{ + {name: "implied", opts: Options{OpenAPPA: true}, wantTimeout: 60 * time.Second}, + {name: "own timeout", opts: Options{OpenAPPA: true, AdditionalEgressExtprocTimeout: "5s"}, wantTimeout: 5 * time.Second}, + { + name: "same service named", + opts: Options{OpenAPPA: true, AdditionalEgressExtprocService: OpenAPPAExtprocService}, + wantTimeout: 60 * time.Second, + }, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + cfg, err := Load(tc.opts) + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if cfg.AdditionalEgressExtprocService != OpenAPPAExtprocService { + t.Errorf("AdditionalEgressExtprocService = %q, want %q", cfg.AdditionalEgressExtprocService, OpenAPPAExtprocService) + } + if !cfg.AdditionalEgressExtprocAllowModeOverride { + t.Error("AdditionalEgressExtprocAllowModeOverride = false, want true") + } + if cfg.AdditionalEgressExtprocTimeout != tc.wantTimeout { + t.Errorf("AdditionalEgressExtprocTimeout = %v, want %v", cfg.AdditionalEgressExtprocTimeout, tc.wantTimeout) + } + if want := []string{"api.githubcopilot.com=github"}; !slices.Equal(cfg.OpenAPPAMCPServers, want) { + t.Errorf("OpenAPPAMCPServers = %v, want %v", cfg.OpenAPPAMCPServers, want) + } + if cfg.OpenAPPARuntimeImage != OpenAPPARuntimeImage || cfg.OpenAPPAExtprocImage != OpenAPPAExtprocImage { + t.Errorf("images = %q, %q, want the defaults", cfg.OpenAPPARuntimeImage, cfg.OpenAPPAExtprocImage) + } + if got := scriptEnvMap(t, cfg)["ATE_OPENAPPA"]; got != "true" { + t.Errorf("ScriptEnv() ATE_OPENAPPA = %q, want true", got) + } + }) + } +} + func TestLoadIngressAllowedSPIFFEIDs(t *testing.T) { const a, b = "spiffe://cluster.local/ns/a/sa/a", "spiffe://cluster.local/ns/b/sa/b" for _, tc := range []struct { @@ -1042,6 +1144,20 @@ func TestLoadRejectsInvalidValues(t *testing.T) { {"cluster size", Options{ClusterSize: "size5"}}, {"extproc invalid format", Options{AdditionalEgressExtprocService: "extproc:50051"}}, {"extproc agentgateway", Options{Router: RouterAgentgateway, AdditionalEgressExtprocService: "ate-system/extproc:50051"}}, + {"extproc mode override without a service", Options{AdditionalEgressExtprocAllowModeOverride: true}}, + {"extproc timeout without a service", Options{AdditionalEgressExtprocTimeout: "5s"}}, + {"extproc timeout zero", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "0s"}}, + {"extproc timeout negative", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "-1s"}}, + {"extproc timeout over an hour", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "61m"}}, + {"openappa on agentgateway", Options{Router: RouterAgentgateway, OpenAPPA: true}}, + {"openappa with another extproc service", Options{OpenAPPA: true, AdditionalEgressExtprocService: "ate-system/extproc:50051"}}, + {"openappa policy that does not exist", Options{OpenAPPA: true, OpenAPPAPolicy: "/nonexistent/appa.toml"}}, + {"openappa policy is a directory", Options{OpenAPPA: true, OpenAPPAPolicy: "."}}, + {"openappa mcp server without a name", Options{OpenAPPA: true, OpenAPPAMCPServers: "api.githubcopilot.com"}}, + {"openappa mcp server with an empty host", Options{OpenAPPA: true, OpenAPPAMCPServers: "=github"}}, + {"openappa mcp server as a URL", Options{OpenAPPA: true, OpenAPPAMCPServers: "https://api.githubcopilot.com=github"}}, + {"openappa policy without openappa", Options{OpenAPPAPolicy: "appa.toml"}}, + {"openappa mcp servers without openappa", Options{OpenAPPAMCPServers: "a.example.com=a"}}, {"provider not JSON", Options{CredentialProvider: "k8s.io"}}, {"provider as a JSON string", Options{CredentialProvider: `"off"`}}, {"provider null", Options{CredentialProvider: `null`}}, diff --git a/cmd/ate-setup/internal/config/coverage_test.go b/cmd/ate-setup/internal/config/coverage_test.go index cbbfa73201..d7f3b03a24 100644 --- a/cmd/ate-setup/internal/config/coverage_test.go +++ b/cmd/ate-setup/internal/config/coverage_test.go @@ -15,9 +15,13 @@ package config import ( + "os" + "path/filepath" "reflect" "strconv" "testing" + + "go.ate.dev/substrate/internal/installdefaults" ) // notFromASetting are Config fields no setting feeds: derived values, the @@ -58,6 +62,12 @@ func TestEveryConfigFieldIsFedBySomeSetting(t *testing.T) { } } + policy := filepath.Join(t.TempDir(), "appa.toml") + if err := os.WriteFile(policy, nil, 0o600); err != nil { + t.Fatal(err) + } + env["ATE_OPENAPPA_POLICY"] = policy + r, err := Resolve(nil, ResolveOptions{Env: env}) if err != nil { t.Fatalf("Resolve() error = %v", err) @@ -95,7 +105,13 @@ func valueFor(key string) string { case "ateapi.postgres.cloudsql.ipType": return CloudSQLIPTypePSC case "atenet.egress.additionalExtprocService": - return "ns/svc:8080" + // openappa.enabled is set too, and refuses any other service. + return OpenAPPAExtprocService + case "openappa.mcpServers": + return "mcp.example.com=example" + case "namespace": + // openappa.enabled is set too, and its manifest names this namespace. + return installdefaults.SystemNamespace case "atenet.egress.credentialProvider": return `{"name":"k8s.io"}` case "actorJWT.algorithm": diff --git a/cmd/ate-setup/internal/config/errors_test.go b/cmd/ate-setup/internal/config/errors_test.go index 87169d6be8..4629b0aa73 100644 --- a/cmd/ate-setup/internal/config/errors_test.go +++ b/cmd/ate-setup/internal/config/errors_test.go @@ -209,6 +209,50 @@ func TestValidationErrorsNameTheSupplyingChannel(t *testing.T) { env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:99999"}, wantIn: []string{"atenet.egress.additionalExtprocService", "1-65535", "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"}, }, + { + name: "extproc mode override without a service", + env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "true"}, + wantIn: []string{"atenet.egress.additionalExtprocAllowModeOverride", "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", "atenet.egress.additionalExtprocService"}, + }, + { + name: "extproc timeout without a service", + env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "5s"}, + wantIn: []string{"atenet.egress.additionalExtprocTimeout", "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT", "atenet.egress.additionalExtprocService"}, + }, + { + name: "extproc timeout is zero", + env: map[string]string{ + "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:50051", + "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "0s", + }, + wantIn: []string{"atenet.egress.additionalExtprocTimeout", "at most 1h", "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT"}, + }, + { + name: "openappa with another extproc service", + env: map[string]string{ + "ATE_OPENAPPA": "true", + "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:50051", + }, + wantIn: []string{"atenet.egress.additionalExtprocService", "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE", "openappa.enabled", "ATE_OPENAPPA", OpenAPPAExtprocService}, + }, + { + name: "openappa with mode overrides turned off", + env: map[string]string{ + "ATE_OPENAPPA": "true", + "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "false", + }, + wantIn: []string{"atenet.egress.additionalExtprocAllowModeOverride", "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", "openappa.enabled"}, + }, + { + name: "openappa in another namespace", + env: map[string]string{"ATE_OPENAPPA": "true", "ATE_NAMESPACE": "elsewhere"}, + wantIn: []string{"openappa.enabled", "namespace", "ATE_NAMESPACE", "manifests/openappa"}, + }, + { + name: "openappa image without openappa", + env: map[string]string{"ATE_OPENAPPA_RUNTIME_IMAGE": "example.com/runtime:v1"}, + wantIn: []string{"openappa.runtimeImage", "ATE_OPENAPPA_RUNTIME_IMAGE", "openappa.enabled"}, + }, { name: "image tag without a repository", env: map[string]string{"ATE_IMAGE_TAG": "v1"}, diff --git a/cmd/ate-setup/internal/config/load.go b/cmd/ate-setup/internal/config/load.go index c9410fab3a..92e237188f 100644 --- a/cmd/ate-setup/internal/config/load.go +++ b/cmd/ate-setup/internal/config/load.go @@ -20,6 +20,7 @@ import ( "path/filepath" "runtime" "strings" + "time" "github.com/spf13/pflag" "github.com/spiffe/go-spiffe/v2/spiffeid" @@ -168,6 +169,11 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro if r.Bool(kindEnabledKey) { applyKindDefaults(r) } + if r.Bool("openappa.enabled") { + if err := applyOpenAPPADefaults(r); err != nil { + return nil, err + } + } rolloutTimeout := r.Duration("rolloutTimeout") if r.Supplied("rolloutTimeout") && rolloutTimeout <= 0 { @@ -262,25 +268,32 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro IAMAuth: r.String("ateapi.postgres.cloudsql.iamAuth"), IPType: r.String("ateapi.postgres.cloudsql.ipType"), }, - RolloutTimeout: rolloutTimeout, - rolloutTimeoutSet: r.Supplied("rolloutTimeout"), - PodcertWorkersPerSigner: podcertWorkers, - EnvoyConcurrency: envoyConcurrency, - ClusterSize: r.String("clusterSize"), - CordonSystemInfraComponents: r.Bool("cordonSystemInfraComponents"), - FakeDataPlane: r.Bool("fakeDataPlane.enabled"), - E2ETuning: r.Bool("e2e.tuning"), - IngressAllowedSPIFFEIDs: splitList(r.String("ingressAllowedSPIFFEIDs")), - AdditionalEgressExtprocService: r.String("atenet.egress.additionalExtprocService"), - CredentialProviderJSON: r.String("atenet.egress.credentialProvider"), - AnthropicAPIKey: r.String("demo.anthropicAPIKey"), - OtlpEndpoint: r.String("otlpEndpoint"), - APILogLevel: r.String("ateapi.logLevel"), - BenchmarkActorMemory: r.String("benchmark.actorMemory"), - PreviewGates: splitList(r.String("preview")), - kubeconfigEnv: kubeconfigEnv, - shellEnv: env, - resolved: r, + RolloutTimeout: rolloutTimeout, + rolloutTimeoutSet: r.Supplied("rolloutTimeout"), + PodcertWorkersPerSigner: podcertWorkers, + EnvoyConcurrency: envoyConcurrency, + ClusterSize: r.String("clusterSize"), + CordonSystemInfraComponents: r.Bool("cordonSystemInfraComponents"), + FakeDataPlane: r.Bool("fakeDataPlane.enabled"), + E2ETuning: r.Bool("e2e.tuning"), + IngressAllowedSPIFFEIDs: splitList(r.String("ingressAllowedSPIFFEIDs")), + AdditionalEgressExtprocService: r.String("atenet.egress.additionalExtprocService"), + AdditionalEgressExtprocAllowModeOverride: r.Bool("atenet.egress.additionalExtprocAllowModeOverride"), + AdditionalEgressExtprocTimeout: r.Duration("atenet.egress.additionalExtprocTimeout"), + OpenAPPA: r.Bool("openappa.enabled"), + OpenAPPAPolicy: r.String("openappa.policy"), + OpenAPPAMCPServers: splitList(r.String("openappa.mcpServers")), + OpenAPPARuntimeImage: r.String("openappa.runtimeImage"), + OpenAPPAExtprocImage: r.String("openappa.extprocImage"), + CredentialProviderJSON: r.String("atenet.egress.credentialProvider"), + AnthropicAPIKey: r.String("demo.anthropicAPIKey"), + OtlpEndpoint: r.String("otlpEndpoint"), + APILogLevel: r.String("ateapi.logLevel"), + BenchmarkActorMemory: r.String("benchmark.actorMemory"), + PreviewGates: splitList(r.String("preview")), + kubeconfigEnv: kubeconfigEnv, + shellEnv: env, + resolved: r, } if err := validateResolved(cfg, r); err != nil { @@ -363,6 +376,35 @@ func applyKindDefaults(r *Resolved) { } } +// applyOpenAPPADefaults points the additional ext_proc filter at OpenAPPA. +// Like applyKindDefaults it writes into the resolved set, so the report and +// the record show what the install uses. A conflicting value a channel +// supplied is refused rather than overridden. +func applyOpenAPPADefaults(r *Resolved) error { + enabled, _ := r.Value("openappa.enabled") + service, _ := r.Value("atenet.egress.additionalExtprocService") + switch { + case !service.Supplied: + r.set(service.Setting.Key, OpenAPPAExtprocService, OriginOpenAPPA) + case service.Raw != OpenAPPAExtprocService: + return &ConflictError{A: service, B: enabled, + Why: "OpenAPPA is the additional ext_proc filter, served by " + OpenAPPAExtprocService} + } + // OpenAPPA reads tools/call bodies, which needs the mode override. + override, _ := r.Value("atenet.egress.additionalExtprocAllowModeOverride") + switch { + case !override.Supplied: + r.set(override.Setting.Key, "true", OriginOpenAPPA) + case !r.Bool(override.Setting.Key): + return &ConflictError{A: override, B: enabled, + Why: "OpenAPPA needs mode overrides to read tools/call bodies"} + } + if !r.Supplied("atenet.egress.additionalExtprocTimeout") { + r.set("atenet.egress.additionalExtprocTimeout", openAPPAExtprocTimeout, OriginOpenAPPA) + } + return nil +} + // namedExplicitly reports whether a flag or configuration file supplied the // setting, as opposed to it arriving from the ambient environment or a default. func namedExplicitly(r *Resolved, key string) bool { @@ -439,8 +481,11 @@ func validateResolved(cfg *Config, r *Resolved) error { issuer, _ := r.Value("actorJWT.issuer") return fmt.Errorf("%s: %w (from %s)", issuer.Setting.Key, err, issuer.From.Describe(issuer.Setting)) } + if err := validateOpenAPPA(cfg, r); err != nil { + return err + } + extproc, _ := r.Value("atenet.egress.additionalExtprocService") if cfg.AdditionalEgressExtprocService != "" { - extproc, _ := r.Value("atenet.egress.additionalExtprocService") if err := validateExtprocService(extproc); err != nil { return err } @@ -449,6 +494,21 @@ func validateResolved(cfg *Config, r *Resolved) error { Why: "an additional ext_proc filter requires dataplane " + RouterEnvoy} } } + allowModeOverride, _ := r.Value("atenet.egress.additionalExtprocAllowModeOverride") + if cfg.AdditionalEgressExtprocAllowModeOverride && cfg.AdditionalEgressExtprocService == "" { + return &ConflictError{A: allowModeOverride, B: extproc, + Why: "only the additional ext_proc filter uses it"} + } + extprocTimeout, _ := r.Value("atenet.egress.additionalExtprocTimeout") + if extprocTimeout.Supplied && cfg.AdditionalEgressExtprocService == "" { + return &ConflictError{A: extprocTimeout, B: extproc, + Why: "only the additional ext_proc filter uses it"} + } + // Envoy rejects a message_timeout above an hour. Zero would time out + // every request at once. + if cfg.AdditionalEgressExtprocTimeout <= 0 || cfg.AdditionalEgressExtprocTimeout > time.Hour { + return &InvalidError{Value: extprocTimeout, Want: "a positive duration of at most 1h"} + } // A gate the components do not know makes each of them refuse to start, so // catch it here rather than as a failed rollout. if err := preview.Verify(cfg.PreviewGates...); err != nil { @@ -475,3 +535,50 @@ func validateResolved(cfg *Config, r *Resolved) error { } return nil } + +// validateOpenAPPA checks the openappa.* settings. They are refused without +// --openappa, as the extproc settings are without a service. +func validateOpenAPPA(cfg *Config, r *Resolved) error { + enabled, _ := r.Value("openappa.enabled") + if !cfg.OpenAPPA { + for _, key := range []string{"openappa.policy", "openappa.mcpServers", "openappa.runtimeImage", "openappa.extprocImage"} { + if v, _ := r.Value(key); v.Supplied { + return &ConflictError{A: v, B: enabled, Why: "only OpenAPPA uses it"} + } + } + return nil + } + if cfg.Router != RouterEnvoy { + dataplane, _ := r.Value("atenet.dataplane") + return &ConflictError{A: enabled, B: dataplane, Why: "OpenAPPA requires dataplane " + RouterEnvoy} + } + if cfg.Namespace != installdefaults.SystemNamespace { + namespace, _ := r.Value("namespace") + return &ConflictError{A: enabled, B: namespace, + Why: "manifests/openappa names " + installdefaults.SystemNamespace + " literally"} + } + if cfg.OpenAPPAPolicy != "" { + policy, _ := r.Value("openappa.policy") + if info, err := os.Stat(cfg.OpenAPPAPolicy); err != nil || !info.Mode().IsRegular() { + return &InvalidError{Value: policy, Want: "a readable appa.toml file"} + } + } + servers, _ := r.Value("openappa.mcpServers") + if len(cfg.OpenAPPAMCPServers) == 0 { + return &InvalidError{Value: servers, Want: openAPPAMCPServersForm} + } + for _, server := range cfg.OpenAPPAMCPServers { + host, name, ok := strings.Cut(server, "=") + if !ok || host == "" || name == "" || strings.ContainsAny(host, "/=") || strings.Contains(name, "=") { + return &InvalidError{Value: servers, Want: openAPPAMCPServersForm} + } + } + for _, key := range []string{"openappa.runtimeImage", "openappa.extprocImage"} { + if v, _ := r.Value(key); v.Raw == "" { + return &InvalidError{Value: v, Want: "an image reference"} + } + } + return nil +} + +const openAPPAMCPServersForm = "comma-separated HOST=NAME pairs, such as api.githubcopilot.com=github" diff --git a/cmd/ate-setup/internal/config/options_test.go b/cmd/ate-setup/internal/config/options_test.go index a6d03c514d..bd9330faf8 100644 --- a/cmd/ate-setup/internal/config/options_test.go +++ b/cmd/ate-setup/internal/config/options_test.go @@ -30,20 +30,25 @@ import ( // Options carries the raw flag values the root command collects, before // defaulting and validation. type Options struct { - Kind bool - Kubeconfig string - Context string - Router string - RolloutTimeout string - PodcertWorkersPerSigner int - EnvoyConcurrency int - ClusterSize string - CordonSystemInfraComponents bool - FakeDataPlane bool - E2ETuning bool - AdditionalEgressExtprocService string - CredentialProvider string - OtlpEndpoint string + Kind bool + Kubeconfig string + Context string + Router string + RolloutTimeout string + PodcertWorkersPerSigner int + EnvoyConcurrency int + ClusterSize string + CordonSystemInfraComponents bool + FakeDataPlane bool + E2ETuning bool + AdditionalEgressExtprocService string + AdditionalEgressExtprocTimeout string + AdditionalEgressExtprocAllowModeOverride bool + OpenAPPA bool + OpenAPPAPolicy string + OpenAPPAMCPServers string + CredentialProvider string + OtlpEndpoint string // Image source selection. ImageRepo string @@ -90,6 +95,9 @@ func optionsFlagSet(opts Options) (*pflag.FlagSet, error) { {"rollout-timeout", opts.RolloutTimeout}, {"cluster-size", opts.ClusterSize}, {"experimental-additional-egress-extproc-service", opts.AdditionalEgressExtprocService}, + {"experimental-additional-egress-extproc-timeout", opts.AdditionalEgressExtprocTimeout}, + {"openappa-policy", opts.OpenAPPAPolicy}, + {"openappa-mcp-servers", opts.OpenAPPAMCPServers}, {"credential-provider", opts.CredentialProvider}, {"otlp-endpoint", opts.OtlpEndpoint}, {"image-repo", opts.ImageRepo}, @@ -107,6 +115,8 @@ func optionsFlagSet(opts Options) (*pflag.FlagSet, error) { {"cordon-system-infra-components", opts.CordonSystemInfraComponents}, {"fake-data-plane", opts.FakeDataPlane}, {"e2e-tuning", opts.E2ETuning}, + {"experimental-additional-egress-extproc-allow-mode-override", opts.AdditionalEgressExtprocAllowModeOverride}, + {"openappa", opts.OpenAPPA}, } { if kv.val { if err := fs.Set(kv.flag, "true"); err != nil { diff --git a/cmd/ate-setup/internal/config/resolve.go b/cmd/ate-setup/internal/config/resolve.go index 5fc1b7237c..0f969d71d0 100644 --- a/cmd/ate-setup/internal/config/resolve.go +++ b/cmd/ate-setup/internal/config/resolve.go @@ -35,6 +35,9 @@ const ( // no channel supplied one. It is not a channel an operator writes to: the // way to override it is any of the three above. OriginKindProfile + // OriginOpenAPPA is an additional ext_proc setting that --openappa implied + // because no channel supplied one. + OriginOpenAPPA ) // Describe names the origin the way the operator would refer to it, so an @@ -49,6 +52,8 @@ func (o Origin) Describe(s Setting) string { return "--" + s.Flag case OriginKindProfile: return "the --kind profile" + case OriginOpenAPPA: + return "--openappa" default: return "default" } @@ -64,6 +69,8 @@ func (o Origin) String() string { return "flag" case OriginKindProfile: return "kind" + case OriginOpenAPPA: + return "openappa" default: return "default" } diff --git a/cmd/ate-setup/internal/config/setting.go b/cmd/ate-setup/internal/config/setting.go index cb3dfd15c8..e8ecbd380d 100644 --- a/cmd/ate-setup/internal/config/setting.go +++ b/cmd/ate-setup/internal/config/setting.go @@ -169,6 +169,46 @@ var Registry = []Setting{ Usage: "Run an additional ext_proc authorization filter served by NS/SVC:PORT " + "(requires --atenet-dataplane=envoy)", }, + { + Key: "atenet.egress.additionalExtprocAllowModeOverride", Env: "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", + Flag: "experimental-additional-egress-extproc-allow-mode-override", Kind: KindBool, Default: "false", + Usage: "Let the additional ext_proc processor buffer the request and response bodies of a request " + + "by returning a mode_override from its request_headers response " + + "(requires --experimental-additional-egress-extproc-service)", + }, + { + Key: "atenet.egress.additionalExtprocTimeout", Env: "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT", + Flag: "experimental-additional-egress-extproc-timeout", Kind: KindDuration, Default: "2s", + Usage: "How long Envoy waits for each additional ext_proc response, at most 1h " + + "(requires --experimental-additional-egress-extproc-service)", + }, + + { + Key: "openappa.enabled", Env: "ATE_OPENAPPA", Flag: "openappa", Kind: KindBool, Default: "false", + Usage: "Deploy OpenAPPA as the egress gateway's additional ext_proc filter " + + "(requires --atenet-dataplane=envoy)", + }, + { + Key: "openappa.policy", Env: "ATE_OPENAPPA_POLICY", Flag: "openappa-policy", Kind: KindString, + Usage: "Path to the OpenAPPA policy, an appa.toml, applied on every deploy " + + "(default: the bundled policy, applied only when the cluster has none)", + }, + { + // Comma-separated; the registry has no list kind. + Key: "openappa.mcpServers", Env: "ATE_OPENAPPA_MCP_SERVERS", Flag: "openappa-mcp-servers", + Kind: KindString, Default: "api.githubcopilot.com=github", + Usage: "Comma-separated HOST=NAME pairs naming the MCP servers OpenAPPA checks tool calls to", + }, + { + Key: "openappa.runtimeImage", Env: "ATE_OPENAPPA_RUNTIME_IMAGE", Flag: "openappa-runtime-image", + Kind: KindString, Default: OpenAPPARuntimeImage, + Usage: "OpenAPPA runtime image", + }, + { + Key: "openappa.extprocImage", Env: "ATE_OPENAPPA_EXTPROC_IMAGE", Flag: "openappa-extproc-image", + Kind: KindString, Default: OpenAPPAExtprocImage, + Usage: "OpenAPPA ext_proc image", + }, { // No default: an unset level keeps the one an existing install has, diff --git a/cmd/ate-setup/internal/steps/delete.go b/cmd/ate-setup/internal/steps/delete.go index 3e52e7ec7a..0da48f91e1 100644 --- a/cmd/ate-setup/internal/steps/delete.go +++ b/cmd/ate-setup/internal/steps/delete.go @@ -176,16 +176,19 @@ func (e *Env) DeleteTrustBundles(ctx context.Context) error { return nil } -// DeleteAtenet removes the atenet dataplane and the bundled credential -// provider, deleting the provider first so that a later failure cannot leave -// its ClusterRole and binding behind. The provider's policy ConfigMap stays: -// it holds the operator's allow-list. +// DeleteAtenet removes the atenet dataplane, the bundled credential provider +// and OpenAPPA, deleting the provider first so that a later failure cannot +// leave its ClusterRole and binding behind. The provider's and OpenAPPA's +// policy ConfigMaps stay: they hold the operator's policy. func (e *Env) DeleteAtenet(ctx context.Context) error { log.Step("delete_atenet") if err := e.Kube.DeletePath(ctx, e.k8sCredentialProviderPath(k8sCredentialProviderManifest)); err != nil { return err } + if err := e.Kube.DeletePath(ctx, e.openAPPAPath(openAPPAManifest)); err != nil { + return err + } for _, path := range [][]string{ {"atenet-router.yaml"}, diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go index b1f9c4fe7a..52e6137e1b 100644 --- a/cmd/ate-setup/internal/steps/deploy.go +++ b/cmd/ate-setup/internal/steps/deploy.go @@ -174,8 +174,8 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err := e.EnsureEgressMITMCAPoolSecret(ctx); err != nil { return err } - // After the podcertificate controller: the provider serves with a - // projected pod certificate. + // After the podcertificate controller: the provider and OpenAPPA serve + // with projected pod certificates. if config.HasKubernetesProvider(providers) { err = e.deployK8sCredentialProvider(ctx) } else { @@ -184,6 +184,14 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if err != nil { return err } + if e.Cfg.OpenAPPA { + err = e.deployOpenAPPA(ctx) + } else { + err = e.removeOpenAPPA(ctx) + } + if err != nil { + return err + } if err := e.applyAtenetEgress(ctx, providers); err != nil { return err } @@ -207,6 +215,9 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { if config.HasKubernetesProvider(providers) { waits = append(waits, rollout{kube.KindDeployment, k8sCredentialProviderDeployment}) } + if e.Cfg.OpenAPPA { + waits = append(waits, rollout{kube.KindDeployment, openAPPADeployment}) + } for _, w := range waits { if err := e.Kube.RolloutStatus(ctx, w.kind, e.Namespace(), w.name, e.Cfg.RolloutTimeout); err != nil { return err @@ -587,6 +598,14 @@ func (e *Env) DeployAtenet(ctx context.Context) error { if err != nil { return err } + if e.Cfg.OpenAPPA { + err = e.deployOpenAPPA(ctx) + } else { + err = e.removeOpenAPPA(ctx) + } + if err != nil { + return err + } if err := e.applyAtenetEgress(ctx, providers); err != nil { return err } @@ -595,6 +614,9 @@ func (e *Env) DeployAtenet(ctx context.Context) error { if config.HasKubernetesProvider(providers) { deployments = append(deployments, k8sCredentialProviderDeployment) } + if e.Cfg.OpenAPPA { + deployments = append(deployments, openAPPADeployment) + } for _, name := range deployments { if err := e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), name, e.Cfg.RolloutTimeout); err != nil { return err diff --git a/cmd/ate-setup/internal/steps/openappa.go b/cmd/ate-setup/internal/steps/openappa.go new file mode 100644 index 0000000000..995cb9265b --- /dev/null +++ b/cmd/ate-setup/internal/steps/openappa.go @@ -0,0 +1,169 @@ +// Copyright 2026 The Agent Substrate Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package steps + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "os" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + + "go.ate.dev/substrate/cmd/ate-setup/internal/kube" + "go.ate.dev/substrate/cmd/ate-setup/internal/log" +) + +// OpenAPPA's manifests. Like the credential provider's, they sit outside the +// ate-install tree because only installs that select it apply them. +const ( + openAPPADir = "manifests/openappa" + openAPPAManifest = "openappa.yaml" + openAPPAPolicy = "appa.toml" + + // Object names from the manifest above; openappa_test.go checks them. + openAPPADeployment = "openappa" + openAPPAPolicyConfigMap = "openappa-policy" + + // openAPPAPolicyHashAnnotation carries a digest of the policy on the pod + // template. The runtime reads the policy at startup, so the digest is what + // turns a changed policy into a restart. + openAPPAPolicyHashAnnotation = "ate.dev/openappa-policy-hash" +) + +func (e *Env) openAPPAPath(name string) string { + return e.Cfg.Path(openAPPADir, name) +} + +// deployOpenAPPA deploys OpenAPPA. The policy ConfigMap is applied from +// --openappa-policy when it is set. Otherwise the bundled policy is applied +// only when the ConfigMap is absent, so a redeploy keeps an operator's edits. +func (e *Env) deployOpenAPPA(ctx context.Context) error { + log.Step("deploy_openappa") + policy, err := e.applyOpenAPPAPolicy(ctx) + if err != nil { + return err + } + manifest, err := e.renderOpenAPPA(policy) + if err != nil { + return err + } + return e.Kube.Apply(ctx, manifest) +} + +// removeOpenAPPA removes OpenAPPA left by an earlier install. The policy +// ConfigMap and the provider credentials Secret stay: the operator may have +// written both. +func (e *Env) removeOpenAPPA(ctx context.Context) error { + running, err := e.Kube.DeploymentExists(ctx, e.Namespace(), openAPPADeployment) + if err != nil || !running { + return err + } + log.Step("remove_openappa") + return e.Kube.DeletePath(ctx, e.openAPPAPath(openAPPAManifest)) +} + +// renderOpenAPPA renders the manifest with the configured images and MCP +// servers, and stamps the Deployment with the policy's digest. +func (e *Env) renderOpenAPPA(policy string) ([]*unstructured.Unstructured, error) { + rendered, err := e.render(e.openAPPAPath(openAPPAManifest)) + if err != nil { + return nil, err + } + objs, err := kube.DecodeManifestBytes(rendered) + if err != nil { + return nil, err + } + dep := findObject(objs, "Deployment", openAPPADeployment) + if dep == nil { + return nil, fmt.Errorf("%s has no deployment/%s", openAPPAManifest, openAPPADeployment) + } + containers, _, err := unstructured.NestedSlice(dep.Object, "spec", "template", "spec", "containers") + if err != nil { + return nil, fmt.Errorf("%s: %w", kube.Describe(dep), err) + } + set := map[string]bool{} + for i, c := range containers { + container := c.(map[string]any) + switch container["name"] { + case "extproc": + container["image"] = e.Cfg.OpenAPPAExtprocImage + args, _, err := unstructured.NestedStringSlice(container, "args") + if err != nil { + return nil, fmt.Errorf("%s: %w", kube.Describe(dep), err) + } + for _, server := range e.Cfg.OpenAPPAMCPServers { + args = append(args, "--mcp-server="+server) + } + if err := unstructured.SetNestedStringSlice(container, args, "args"); err != nil { + return nil, fmt.Errorf("%s: %w", kube.Describe(dep), err) + } + case "runtime": + container["image"] = e.Cfg.OpenAPPARuntimeImage + default: + continue + } + set[container["name"].(string)] = true + containers[i] = container + } + if !set["extproc"] || !set["runtime"] { + return nil, fmt.Errorf("%s needs extproc and runtime containers", kube.Describe(dep)) + } + if err := unstructured.SetNestedSlice(dep.Object, containers, "spec", "template", "spec", "containers"); err != nil { + return nil, err + } + sum := sha256.Sum256([]byte(policy)) + if err := unstructured.SetNestedField(dep.Object, hex.EncodeToString(sum[:]), + "spec", "template", "metadata", "annotations", openAPPAPolicyHashAnnotation); err != nil { + return nil, err + } + return objs, nil +} + +// applyOpenAPPAPolicy reconciles the policy ConfigMap and returns the policy +// it holds. +func (e *Env) applyOpenAPPAPolicy(ctx context.Context) (string, error) { + path := e.Cfg.OpenAPPAPolicy + if path == "" { + cm, err := e.Kube.GetConfigMap(ctx, e.Namespace(), openAPPAPolicyConfigMap) + if err != nil { + return "", err + } + if cm != nil { + return cm.Data[openAPPAPolicy], nil + } + path = e.openAPPAPath(openAPPAPolicy) + } + policy, err := os.ReadFile(path) + if err != nil { + return "", fmt.Errorf("while reading the OpenAPPA policy: %w", err) + } + if err := e.Kube.ApplyConfigMap(ctx, e.Namespace(), openAPPAPolicyConfigMap, + map[string]string{openAPPAPolicy: string(policy)}); err != nil { + return "", err + } + return string(policy), nil +} + +// findObject returns the named object of a kind, or nil. +func findObject(objs []*unstructured.Unstructured, kind, name string) *unstructured.Unstructured { + for _, obj := range objs { + if obj.GetKind() == kind && obj.GetName() == name { + return obj + } + } + return nil +} diff --git a/cmd/ate-setup/internal/steps/openappa_test.go b/cmd/ate-setup/internal/steps/openappa_test.go new file mode 100644 index 0000000000..79835e92f8 --- /dev/null +++ b/cmd/ate-setup/internal/steps/openappa_test.go @@ -0,0 +1,284 @@ +// Copyright 2026 The Agent Substrate Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package steps + +import ( + "context" + "os" + "path/filepath" + "slices" + "strconv" + "strings" + "testing" + + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/kubernetes/fake" + + "go.ate.dev/substrate/cmd/ate-setup/internal/config" + "go.ate.dev/substrate/cmd/ate-setup/internal/kube" +) + +// The deploy step and the egress filter refer to OpenAPPA's objects by name +// and address. This checks them against the manifests, so a rename fails here +// rather than in the next install. +func TestOpenAPPAManifestAgrees(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}} + objs, err := kube.LoadPath(e.openAPPAPath(openAPPAManifest)) + if err != nil { + t.Fatalf("loading the OpenAPPA manifest: %v", err) + } + for _, obj := range objs { + if obj.GetNamespace() != NamespaceAteSystem { + t.Errorf("%s is in namespace %q, want %q", kube.Describe(obj), obj.GetNamespace(), NamespaceAteSystem) + } + } + dep := findObject(objs, "Deployment", openAPPADeployment) + if dep == nil { + t.Fatalf("manifest has no deployment/%s", openAPPADeployment) + } + if findObject(objs, "ConfigMap", openAPPAPolicyConfigMap) != nil { + t.Errorf("manifest carries configmap/%s itself; the deploy step must own whether it is applied", openAPPAPolicyConfigMap) + } + if got := mountedConfigMaps(dep); len(got) != 1 || got[0] != openAPPAPolicyConfigMap { + t.Errorf("deployment/%s mounts ConfigMaps %v, want only %q", openAPPADeployment, got, openAPPAPolicyConfigMap) + } + svc := findObject(objs, "Service", openAPPADeployment) + if svc == nil { + t.Fatalf("manifest has no service/%s", openAPPADeployment) + } + ports, _, _ := unstructured.NestedSlice(svc.Object, "spec", "ports") + if len(ports) != 1 { + t.Fatalf("service/%s has %d ports, want 1", openAPPADeployment, len(ports)) + } + port, _ := asInt(ports[0].(map[string]any)["port"]) + if want := svc.GetNamespace() + "/" + svc.GetName() + ":" + strconv.FormatInt(port, 10); want != config.OpenAPPAExtprocService { + t.Errorf("config.OpenAPPAExtprocService = %q, but the Service is %q", config.OpenAPPAExtprocService, want) + } + if got := containerPort(t, dep, "grpc"); got != port { + t.Errorf("the extproc container serves on %d, but the Service forwards to %d", got, port) + } + + // The processor admits only the egress gateway's identity. + egress, err := kube.LoadPath(e.atenetEgressManifestPath()) + if err != nil { + t.Fatalf("loading the egress manifest: %v", err) + } + gateway := findObject(egress, "Deployment", "atenet-egress") + if gateway == nil { + t.Fatal("egress manifest has no deployment/atenet-egress") + } + sa, _, _ := unstructured.NestedString(gateway.Object, "spec", "template", "spec", "serviceAccountName") + wantID := "--tls-client-id=spiffe://cluster.local/ns/" + gateway.GetNamespace() + "/sa/" + sa + if args := deploymentContainerArgs(t, dep, "extproc"); !slices.Contains(args, wantID) { + t.Errorf("extproc args %v lack %s", args, wantID) + } +} + +func TestRenderOpenAPPA(t *testing.T) { + for _, tc := range []struct { + name string + cfg config.Config + wantServers []string + }{ + { + name: "defaults", + cfg: config.Config{ + OpenAPPAMCPServers: []string{"api.githubcopilot.com=github"}, + OpenAPPARuntimeImage: config.OpenAPPARuntimeImage, + OpenAPPAExtprocImage: config.OpenAPPAExtprocImage, + }, + wantServers: []string{"--mcp-server=api.githubcopilot.com=github"}, + }, + { + name: "overrides", + cfg: config.Config{ + OpenAPPAMCPServers: []string{"a.example.com=a", "b.example.com=b"}, + OpenAPPARuntimeImage: "example.com/runtime:v1", + OpenAPPAExtprocImage: "example.com/extproc:v1", + }, + wantServers: []string{"--mcp-server=a.example.com=a", "--mcp-server=b.example.com=b"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := tc.cfg + cfg.Root = repoRoot(t) + e := &Env{Cfg: &cfg} + objs, err := e.renderOpenAPPA("policy") + if err != nil { + t.Fatalf("renderOpenAPPA: %v", err) + } + dep := findObject(objs, "Deployment", openAPPADeployment) + if got := deploymentContainerImage(t, dep, "extproc"); got != cfg.OpenAPPAExtprocImage { + t.Errorf("extproc image = %q, want %q", got, cfg.OpenAPPAExtprocImage) + } + if got := deploymentContainerImage(t, dep, "runtime"); got != cfg.OpenAPPARuntimeImage { + t.Errorf("runtime image = %q, want %q", got, cfg.OpenAPPARuntimeImage) + } + var servers []string + for _, arg := range deploymentContainerArgs(t, dep, "extproc") { + if strings.HasPrefix(arg, "--mcp-server=") { + servers = append(servers, arg) + } + } + if !slices.Equal(servers, tc.wantServers) { + t.Errorf("extproc --mcp-server args = %v, want %v", servers, tc.wantServers) + } + }) + } +} + +// A changed policy changes the pod template, so the runtime restarts and +// reads it. +func TestRenderOpenAPPAStampsPolicyHash(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}} + hash := func(policy string) string { + t.Helper() + objs, err := e.renderOpenAPPA(policy) + if err != nil { + t.Fatalf("renderOpenAPPA: %v", err) + } + dep := findObject(objs, "Deployment", openAPPADeployment) + got, _, _ := unstructured.NestedString(dep.Object, "spec", "template", "metadata", "annotations", openAPPAPolicyHashAnnotation) + if got == "" { + t.Fatalf("deployment/%s has no %s", openAPPADeployment, openAPPAPolicyHashAnnotation) + } + return got + } + if hash("a") == hash("b") { + t.Error("two policies stamp the same digest") + } +} + +// The bundled policy fills a cluster that has none and never replaces one an +// operator edited. A policy file always replaces it. +func TestApplyOpenAPPAPolicy(t *testing.T) { + root := repoRoot(t) + bundled, err := os.ReadFile(filepath.Join(root, openAPPADir, openAPPAPolicy)) + if err != nil { + t.Fatal(err) + } + file := filepath.Join(t.TempDir(), "appa.toml") + if err := os.WriteFile(file, []byte("from file"), 0o600); err != nil { + t.Fatal(err) + } + edited := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: openAPPAPolicyConfigMap}, + Data: map[string]string{openAPPAPolicy: "edited"}, + } + for _, tc := range []struct { + name string + existing runtime.Object + policy string + want string + }{ + {name: "fresh install gets the bundled policy", want: string(bundled)}, + {name: "redeploy keeps an edited policy", existing: edited, want: "edited"}, + {name: "fresh install with a file", policy: file, want: "from file"}, + {name: "a file replaces an edited policy", existing: edited.DeepCopy(), policy: file, want: "from file"}, + } { + t.Run(tc.name, func(t *testing.T) { + var existing []runtime.Object + if tc.existing != nil { + existing = append(existing, tc.existing) + } + // NewClientset rather than fakeKube: only it supports apply. + k := &kube.Client{Typed: fake.NewClientset(existing...)} + e := &Env{Cfg: &config.Config{Root: root, OpenAPPAPolicy: tc.policy}, Kube: k} + got, err := e.applyOpenAPPAPolicy(context.Background()) + if err != nil { + t.Fatalf("applyOpenAPPAPolicy: %v", err) + } + if got != tc.want { + t.Errorf("applyOpenAPPAPolicy() = %q, want %q", got, tc.want) + } + cm, err := e.Kube.GetConfigMap(context.Background(), NamespaceAteSystem, openAPPAPolicyConfigMap) + if err != nil || cm == nil { + t.Fatalf("GetConfigMap = %v, %v", cm, err) + } + if cm.Data[openAPPAPolicy] != tc.want { + t.Errorf("configmap/%s holds %q, want %q", openAPPAPolicyConfigMap, cm.Data[openAPPAPolicy], tc.want) + } + }) + } +} + +// A redeploy without --openappa removes what an earlier one deployed, and +// keeps the operator's policy. +func TestRemoveOpenAPPA(t *testing.T) { + services := schema.GroupVersionResource{Version: "v1", Resource: "services"} + for _, tc := range []struct { + name string + running bool + }{ + {name: "earlier install had it", running: true}, + {name: "nothing to remove"}, + } { + t.Run(tc.name, func(t *testing.T) { + policy := &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: openAPPAPolicyConfigMap}} + typed := []runtime.Object{policy} + var dyn []runtime.Object + if tc.running { + typed = append(typed, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: openAPPADeployment}}) + dyn = append(dyn, &corev1.Service{ObjectMeta: metav1.ObjectMeta{Namespace: NamespaceAteSystem, Name: openAPPADeployment}}) + } + k, dc := fakeDeleteKube(t, typed, dyn) + e := &Env{Cfg: &config.Config{Root: repoRoot(t)}, Kube: k} + if err := e.removeOpenAPPA(context.Background()); err != nil { + t.Fatalf("removeOpenAPPA() = %v", err) + } + if tc.running { + _, err := dc.Resource(services).Namespace(NamespaceAteSystem).Get(context.Background(), openAPPADeployment, metav1.GetOptions{}) + if !apierrors.IsNotFound(err) { + t.Errorf("service/%s survived removeOpenAPPA (err %v)", openAPPADeployment, err) + } + } + if cm, err := k.GetConfigMap(context.Background(), NamespaceAteSystem, openAPPAPolicyConfigMap); err != nil || cm == nil { + t.Errorf("configmap/%s is gone after removeOpenAPPA (err %v)", openAPPAPolicyConfigMap, err) + } + }) + } +} + +// deploymentContainerArgs returns the args of a Deployment's named container. +func deploymentContainerArgs(t *testing.T, dep *unstructured.Unstructured, name string) []string { + t.Helper() + args, _, _ := unstructured.NestedStringSlice(deploymentContainer(t, dep, name), "args") + return args +} + +// deploymentContainerImage returns the image of a Deployment's named container. +func deploymentContainerImage(t *testing.T, dep *unstructured.Unstructured, name string) string { + t.Helper() + image, _, _ := unstructured.NestedString(deploymentContainer(t, dep, name), "image") + return image +} + +func deploymentContainer(t *testing.T, dep *unstructured.Unstructured, name string) map[string]any { + t.Helper() + containers, _, _ := unstructured.NestedSlice(dep.Object, "spec", "template", "spec", "containers") + for _, c := range containers { + if c.(map[string]any)["name"] == name { + return c.(map[string]any) + } + } + t.Fatalf("deployment/%s has no container %q", dep.GetName(), name) + return nil +} diff --git a/cmd/ate-setup/internal/steps/overlay.go b/cmd/ate-setup/internal/steps/overlay.go index 2fbaf8ca66..e659a2cc99 100644 --- a/cmd/ate-setup/internal/steps/overlay.go +++ b/cmd/ate-setup/internal/steps/overlay.go @@ -441,7 +441,7 @@ func (e *Env) patchAtenetEgressManifest() ([]byte, error) { address := fmt.Sprintf("%s.%s.svc.cluster.local", service, namespace) serverName := fmt.Sprintf("%s.%s.svc", service, namespace) - filterBlock := emitAdditionalEgressExtprocFilter() + filterBlock := emitAdditionalEgressExtprocFilter(e.Cfg.AdditionalEgressExtprocTimeout, e.Cfg.AdditionalEgressExtprocAllowModeOverride) clusterBlock := emitAdditionalEgressExtprocCluster(address, port, serverName) lines := strings.Split(string(raw), "\n") @@ -488,16 +488,36 @@ func (e *Env) patchAtenetEgressManifest() ([]byte, error) { const additionalEgressExtprocCluster = "additional_egress_ext_proc" -func emitAdditionalEgressExtprocFilter() string { - return `- name: envoy.filters.http.ext_proc +// emitAdditionalEgressExtprocFilter renders the filter. The processor sees +// request headers only, unless allowModeOverride lets it switch a request to +// buffered bodies and response headers with a mode_override. +func emitAdditionalEgressExtprocFilter(timeout time.Duration, allowModeOverride bool) string { + // A protobuf Duration in YAML is seconds with an "s" suffix; Envoy rejects + // Go's "1m30s" or "500ms". + t := strconv.FormatFloat(timeout.Seconds(), 'f', -1, 64) + "s" + modeOverride := "" + if allowModeOverride { + // Envoy compares the effective mode, the override laid over + // processing_mode, against each entry. It ignores request_header_mode. + modeOverride = ` + allow_mode_override: true + allowed_override_modes: + - request_header_mode: SEND + response_header_mode: SEND + request_body_mode: BUFFERED + response_body_mode: BUFFERED + request_trailer_mode: SKIP + response_trailer_mode: SKIP` + } + return fmt.Sprintf(`- name: envoy.filters.http.ext_proc typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor grpc_service: envoy_grpc: cluster_name: additional_egress_ext_proc - timeout: 2s + timeout: %s failure_mode_allow: false - message_timeout: 2s + message_timeout: %s request_attributes: - filter_state['dev.ate.actor.identity'] processing_mode: @@ -506,10 +526,10 @@ func emitAdditionalEgressExtprocFilter() string { request_body_mode: NONE response_body_mode: NONE request_trailer_mode: SKIP - response_trailer_mode: SKIP + response_trailer_mode: SKIP%s mutation_rules: disallow_system: true - disallow_is_error: true` + disallow_is_error: true`, t, t, modeOverride) } func emitAdditionalEgressExtprocCluster(address, port, serverName string) string { diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go index ab735825d3..e39a512d02 100644 --- a/cmd/ate-setup/internal/steps/overlay_test.go +++ b/cmd/ate-setup/internal/steps/overlay_test.go @@ -16,7 +16,9 @@ package steps import ( "context" + "encoding/json" "fmt" + "maps" "os" "reflect" "slices" @@ -24,6 +26,9 @@ import ( "testing" "time" + extprocv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/ext_proc/v3" + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" @@ -193,6 +198,111 @@ func TestEmitAdditionalEgressExtprocCluster(t *testing.T) { } } +// The defaults render the filter as it was before the timeout and the mode +// override were configurable. Every case must parse as the vendored +// ExternalProcessor, since CI never deploys an additional processor. +func TestEmitAdditionalEgressExtprocFilter(t *testing.T) { + const headersOnly = `- name: envoy.filters.http.ext_proc + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor + grpc_service: + envoy_grpc: + cluster_name: additional_egress_ext_proc + timeout: 2s + failure_mode_allow: false + message_timeout: 2s + request_attributes: + - filter_state['dev.ate.actor.identity'] + processing_mode: + request_header_mode: SEND + response_header_mode: SKIP + request_body_mode: NONE + response_body_mode: NONE + request_trailer_mode: SKIP + response_trailer_mode: SKIP + mutation_rules: + disallow_system: true + disallow_is_error: true` + buffered := &extprocv3.ProcessingMode{ + RequestHeaderMode: extprocv3.ProcessingMode_SEND, + ResponseHeaderMode: extprocv3.ProcessingMode_SEND, + RequestBodyMode: extprocv3.ProcessingMode_BUFFERED, + ResponseBodyMode: extprocv3.ProcessingMode_BUFFERED, + RequestTrailerMode: extprocv3.ProcessingMode_SKIP, + ResponseTrailerMode: extprocv3.ProcessingMode_SKIP, + } + for _, tc := range []struct { + name string + timeout time.Duration + allowModeOverride bool + wantYAML string + wantTimeout time.Duration + wantOverrides []*extprocv3.ProcessingMode + }{ + {name: "defaults", timeout: 2 * time.Second, wantYAML: headersOnly, wantTimeout: 2 * time.Second}, + {name: "fractional timeout", timeout: 1500 * time.Millisecond, wantTimeout: 1500 * time.Millisecond}, + {name: "timeout over a minute", timeout: 90 * time.Second, wantTimeout: 90 * time.Second}, + { + name: "mode override", timeout: 2 * time.Second, allowModeOverride: true, + wantTimeout: 2 * time.Second, wantOverrides: []*extprocv3.ProcessingMode{buffered}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + out := emitAdditionalEgressExtprocFilter(tc.timeout, tc.allowModeOverride) + if tc.wantYAML != "" && out != tc.wantYAML { + t.Errorf("filter changed:\n%s\nwant:\n%s", out, tc.wantYAML) + } + + var filters []map[string]any + if err := yaml.Unmarshal([]byte(out), &filters); err != nil { + t.Fatalf("emitted filter is not valid YAML: %v\n%s", err, out) + } + if len(filters) != 1 { + t.Fatalf("expected 1 filter, got %d", len(filters)) + } + got := externalProcessor(t, filters[0]) + if d := got.GetMessageTimeout().AsDuration(); d != tc.wantTimeout { + t.Errorf("message_timeout = %v, want %v", d, tc.wantTimeout) + } + if d := got.GetGrpcService().GetTimeout().AsDuration(); d != tc.wantTimeout { + t.Errorf("grpc_service.timeout = %v, want %v", d, tc.wantTimeout) + } + if got.GetAllowModeOverride() != tc.allowModeOverride { + t.Errorf("allow_mode_override = %v, want %v", got.GetAllowModeOverride(), tc.allowModeOverride) + } + if !slices.EqualFunc(got.GetAllowedOverrideModes(), tc.wantOverrides, func(a, b *extprocv3.ProcessingMode) bool { + return proto.Equal(a, b) + }) { + t.Errorf("allowed_override_modes = %v, want %v", got.GetAllowedOverrideModes(), tc.wantOverrides) + } + }) + } +} + +// externalProcessor parses an ext_proc filter's typed_config as the vendored +// ExternalProcessor proto and runs its validation rules. +func externalProcessor(t *testing.T, filter map[string]any) *extprocv3.ExternalProcessor { + t.Helper() + typedConfig, _ := filter["typed_config"].(map[string]any) + if typedConfig == nil { + t.Fatal("filter has no typed_config") + } + fields := maps.Clone(typedConfig) + delete(fields, "@type") + raw, err := json.Marshal(fields) + if err != nil { + t.Fatalf("marshaling typed_config: %v", err) + } + var ep extprocv3.ExternalProcessor + if err := protojson.Unmarshal(raw, &ep); err != nil { + t.Fatalf("typed_config is not an ExternalProcessor: %v\n%s", err, raw) + } + if err := ep.ValidateAll(); err != nil { + t.Fatalf("ExternalProcessor fails validation: %v", err) + } + return &ep +} + // Splices the real egress manifest and re-parses the result. CI never deploys // with the extproc flag, so this is the only automated check on the injected // cluster. @@ -202,8 +312,10 @@ func TestPatchAtenetEgressManifest(t *testing.T) { t.Fatalf("resolving repo root: %v", err) } env := &Env{Cfg: &config.Config{ - Root: root, - AdditionalEgressExtprocService: "ate-system/foo:50051", + Root: root, + AdditionalEgressExtprocService: "ate-system/foo:50051", + AdditionalEgressExtprocTimeout: 2 * time.Second, + AdditionalEgressExtprocAllowModeOverride: true, }} patched, err := env.patchAtenetEgressManifest() @@ -292,6 +404,10 @@ func TestPatchAtenetEgressManifest(t *testing.T) { policyAt = i case additionalEgressExtprocCluster: additionalAt = i + // The splice indents the block; it must still parse. + if !externalProcessor(t, f).GetAllowModeOverride() { + t.Error("spliced additional ext_proc lost allow_mode_override") + } } } if additionalAt < 0 { @@ -504,6 +620,11 @@ func TestRenderCordonSystemInfraComponents(t *testing.T) { path: func(e *Env) string { return e.k8sCredentialProviderPath(k8sCredentialProviderManifest) }, want: []string{k8sCredentialProviderDeployment}, }, + { + name: "OpenAPPA file", + path: func(e *Env) string { return e.openAPPAPath(openAPPAManifest) }, + want: []string{openAPPADeployment}, + }, } { t.Run(tc.name, func(t *testing.T) { cfg := tc.cfg @@ -546,6 +667,7 @@ func TestRenderBytesCordonSystemInfraComponents(t *testing.T) { Root: repoRoot(t), CordonSystemInfraComponents: true, AdditionalEgressExtprocService: "ate-system/foo:50051", + AdditionalEgressExtprocTimeout: 2 * time.Second, }} patched, err := e.patchAtenetEgressManifest() if err != nil { @@ -1759,14 +1881,18 @@ func TestRequireInstalledDataPlane(t *testing.T) { } // fakeDeleteKube is fakeKube with a dynamic client and discovery for the -// built-in kinds in fake-workersync's manifest, so DeletePath runs. typed +// built-in kinds in fake-workersync's and OpenAPPA's manifests, so DeletePath +// runs. typed // seeds the typed client, which the Deployment checks read; dyn seeds the // dynamic client, which DeletePath deletes through. func fakeDeleteKube(t *testing.T, typed []runtime.Object, dyn []runtime.Object) (*kube.Client, *dynamicfake.FakeDynamicClient) { t.Helper() cs := fake.NewClientset(typed...) cs.Fake.Resources = []*metav1.APIResourceList{ - {GroupVersion: "v1", APIResources: []metav1.APIResource{{Name: "serviceaccounts", Kind: "ServiceAccount", Namespaced: true}}}, + {GroupVersion: "v1", APIResources: []metav1.APIResource{ + {Name: "serviceaccounts", Kind: "ServiceAccount", Namespaced: true}, + {Name: "services", Kind: "Service", Namespaced: true}, + }}, {GroupVersion: "apps/v1", APIResources: []metav1.APIResource{{Name: "deployments", Kind: "Deployment", Namespaced: true}}}, {GroupVersion: "networking.k8s.io/v1", APIResources: []metav1.APIResource{{Name: "networkpolicies", Kind: "NetworkPolicy", Namespaced: true}}}, {GroupVersion: "rbac.authorization.k8s.io/v1", APIResources: []metav1.APIResource{ diff --git a/cmd/ate-setup/internal/steps/postgres_test.go b/cmd/ate-setup/internal/steps/postgres_test.go index 7a956b9ad7..5d812ea5b0 100644 --- a/cmd/ate-setup/internal/steps/postgres_test.go +++ b/cmd/ate-setup/internal/steps/postgres_test.go @@ -48,15 +48,6 @@ func postgresObjects(t *testing.T, kind bool) []*unstructured.Unstructured { return objs } -func findObject(objs []*unstructured.Unstructured, kind, name string) *unstructured.Unstructured { - for _, obj := range objs { - if obj.GetKind() == kind && obj.GetName() == name { - return obj - } - } - return nil -} - // The kind overlay has to be what a kind install applies: the base file asks // for more CPU than a laptop cluster schedules. func TestPostgresManifestPathSelectsKindOverlay(t *testing.T) { diff --git a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go index 1471c398e4..64bb1a812f 100644 --- a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go +++ b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go @@ -42,7 +42,8 @@ type systemPod struct { // systemInstall is everything an install of one configuration puts in // ate-system: the system overlay plus the manifests the installer applies on -// their own (postgres, the egress gateway, the bundled credential provider). +// their own (postgres, the egress gateway, the bundled credential provider, +// OpenAPPA). type systemInstall struct { pods []systemPod policies []networkingv1.NetworkPolicy @@ -133,6 +134,11 @@ func loadSystemInstall(t *testing.T, cfg *config.Config) systemInstall { t.Fatalf("loading the credential provider manifest: %v", err) } objs = append(objs, provider...) + openappa, err := kube.LoadPath(e.openAPPAPath(openAPPAManifest)) + if err != nil { + t.Fatalf("loading the OpenAPPA manifest: %v", err) + } + objs = append(objs, openappa...) var install systemInstall for _, obj := range objs { @@ -447,6 +453,9 @@ func TestSystemNetworkPoliciesLeastPrivilege(t *testing.T) { atelet := in.pod(t, "atelet") requireOnlyFromApps(t, in, atelet, namedPort(t, atelet, "grpc"), "ate-api-server") + + openappa := in.pod(t, openAPPADeployment) + requireOnlyFromApps(t, in, openappa, namedPort(t, openappa, "grpc"), "atenet-egress") if cfg.FakeDataPlane { requireOnlyFromApps(t, in, atelet, namedPort(t, atelet, "relay"), "fake-workersync") } diff --git a/docs/guides/openappa.md b/docs/guides/openappa.md new file mode 100644 index 0000000000..024dcfa7cd --- /dev/null +++ b/docs/guides/openappa.md @@ -0,0 +1,72 @@ +--- +title: "OpenAPPA" +linkTitle: "OpenAPPA" +weight: 19 +description: > + Check actors' MCP tool calls against what they have already read, at the egress gateway. +--- + +# OpenAPPA + +[OpenAPPA](https://github.com/archestra-ai/OpenAPPA) is an information-flow +policy engine. With `--openappa`, `ate-setup` deploys it as the egress gateway's +additional `ext_proc` filter (see [Extending Substrate](../security/extensibility.md)). +It reads the bodies of the MCP `tools/call` requests an actor sends, and their +responses. It refuses a call that would send data to someone who may not +already see it. See the [OpenAPPA docs](https://www.openappa.com/docs/substrate) +for how the policy works. + +## Enable it + +On an installed cluster, redeploy the dataplane with it: + +```sh +go run ./cmd/ate-setup deploy atenet --credential-provider='{"name":"k8s.io"}' --openappa +``` + +`--openappa` works on `deploy ate-system` and `deploy atenet`, on the envoy +dataplane only. It deploys `deployment/openappa` into `ate-system`, with a +Service and a NetworkPolicy that admits only the egress gateway. It also points +the additional ext_proc filter at that Service, lets the filter buffer bodies, +and sets its timeout to 60s unless +`--experimental-additional-egress-extproc-timeout` is given. Naming a different `--experimental-additional-egress-extproc-service` +is refused. + +A redeploy without `--openappa` removes OpenAPPA. It keeps the policy +ConfigMap and the credentials Secret. + +## Policy + +The policy is `appa.toml` in `configmap/openappa-policy`. + +- Without `--openappa-policy`, the first deploy installs the bundled policy, + [`manifests/openappa/appa.toml`](../../manifests/openappa/appa.toml). It + includes OpenAPPA's GitHub battery. Later deploys leave the ConfigMap + alone, so you can edit it in place. +- With `--openappa-policy=PATH`, every deploy replaces the ConfigMap with that + file. + +A deploy restarts OpenAPPA when the policy changed. + +## MCP servers and egress policy + +`--openappa-mcp-servers` lists the MCP servers OpenAPPA checks, as +`HOST=NAME` pairs. The default is `api.githubcopilot.com=github`. + +OpenAPPA only sees requests the gateway can read. Allow each MCP host in the +actor's `EgressPolicy` with an `https` rule, not `tls_passthrough` (see +[Egress Traffic](egress-traffic.md)). Allow only the MCP hosts, so the actor +cannot send data around OpenAPPA. + +## Provider credentials + +The GitHub battery calls the GitHub API to learn who may read a repository. +Give it a token in the optional `openappa-provider-credentials` Secret: + +```sh +kubectl -n ate-system create secret generic openappa-provider-credentials \ + --from-literal=APPA_PROVIDER_GITHUB_TOKEN= +kubectl -n ate-system rollout restart deployment/openappa +``` + +The runtime reads it at startup, so restart it after a change. diff --git a/docs/guides/operator-install.md b/docs/guides/operator-install.md index c62050a5c0..25473fb66c 100644 --- a/docs/guides/operator-install.md +++ b/docs/guides/operator-install.md @@ -298,7 +298,9 @@ demos, the [benchmarks](benchmarking.md#settings), and the | `ateapi.postgres.serverCAFile` | `ATE_API_POSTGRES_SERVER_CA_FILE` | `--ateapi-postgres-server-ca-file` | — | (global) | | `ateapi.postgres.storageClass` | `ATE_API_POSTGRES_STORAGE_CLASS` | `--ateapi-postgres-storage-class` | — | (global) | | `atenet.dataplane` | `ATE_ATENET_DATAPLANE` | `--atenet-dataplane` | `envoy` | (global) | +| `atenet.egress.additionalExtprocAllowModeOverride` | `ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE` | `--experimental-additional-egress-extproc-allow-mode-override` | `false` | (global) | | `atenet.egress.additionalExtprocService` | `ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE` | `--experimental-additional-egress-extproc-service` | — | (global) | +| `atenet.egress.additionalExtprocTimeout` | `ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT` | `--experimental-additional-egress-extproc-timeout` | `2s` | (global) | | `atenet.egress.credentialProvider` | `ATE_CREDENTIAL_PROVIDER` | `--credential-provider` | — | (global) | | `clusterSize` | `ATE_INSTALL_CLUSTER_SIZE` | `--cluster-size` | `size0` | (global) | | `context` | `KUBECTL_CONTEXT` | `--context` | — | (global) | @@ -318,6 +320,11 @@ demos, the [benchmarks](benchmarking.md#settings), and the | `ko.dockerRepo` | `KO_DOCKER_REPO` | `--ko-docker-repo` | — | (global) | | `kubeconfig` | `KUBECONFIG` | `--kubeconfig` | — | (global) | | `namespace` | `ATE_NAMESPACE` | `--namespace` | `ate-system` | (global) | +| `openappa.enabled` | `ATE_OPENAPPA` | `--openappa` | `false` | (global) | +| `openappa.extprocImage` | `ATE_OPENAPPA_EXTPROC_IMAGE` | `--openappa-extproc-image` | `ghcr.io/archestra-ai/appa-extproc:v0.43.0` | (global) | +| `openappa.mcpServers` | `ATE_OPENAPPA_MCP_SERVERS` | `--openappa-mcp-servers` | `api.githubcopilot.com=github` | (global) | +| `openappa.policy` | `ATE_OPENAPPA_POLICY` | `--openappa-policy` | — (bundled policy) | (global) | +| `openappa.runtimeImage` | `ATE_OPENAPPA_RUNTIME_IMAGE` | `--openappa-runtime-image` | `ghcr.io/archestra-ai/appa-runtime:v0.42.0@sha256:…` | (global) | | `otlpEndpoint` | `ATE_OTLP_ENDPOINT` | `--otlp-endpoint` | — | (global) | | `podcert.workersPerSigner` | `ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER` | `--podcert-workers-per-signer` | `0` | (global) | | `preview` | `ATE_PREVIEW` | `--preview` | — | (global) | diff --git a/docs/security/extensibility.md b/docs/security/extensibility.md index 65b4faa1e7..68a7ef65b1 100644 --- a/docs/security/extensibility.md +++ b/docs/security/extensibility.md @@ -19,7 +19,10 @@ Substrate ships with built-in sandboxing, actor identity, and ingress and egress You can add your own [`ext_proc`](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter) service. Substrate gateways call it on every request. -- **Egress:** set `--experimental-additional-egress-extproc-service` (see [Operator install](../guides/operator-install.md#settings)). This may be converted to an API in the future. Your service runs after Substrate has authenticated the actor and applied its `EgressPolicy`. It currently receives the request headers and the actor's identity, and it can deny the request or add headers; this can be changed if necessary. Today it sees headers only, not bodies. +- **Egress:** set `--experimental-additional-egress-extproc-service` (see [Operator install](../guides/operator-install.md#settings)). This may be converted to an API in the future. Your service runs after Substrate has authenticated the actor and applied its `EgressPolicy`. It receives the request headers and the actor's identity, and it can deny the request or add headers. Envoy waits 2s for each response from it; `--experimental-additional-egress-extproc-timeout` changes that. + - **Bodies:** with `--experimental-additional-egress-extproc-allow-mode-override`, your service can read the request and response bodies of the requests it picks. To pick one, return a `mode_override` in the `request_headers` response with `response_header_mode: SEND`, `request_body_mode: BUFFERED` and `response_body_mode: BUFFERED`. Envoy ignores any other override. It then sends the response headers, the whole request body and the whole response body, and your service can change them too. + - Envoy holds a buffered body until it ends, and fails the request if the body outgrows its buffer limit. Pick only requests whose bodies end, such as an MCP `tools/call` POST. A long-lived stream such as an SSE `GET` never finishes, so leave those in the default mode. + - For example, [OpenAPPA](https://github.com/archestra-ai/OpenAPPA) buffers each MCP `tools/call` this way. It checks the call against the data the actor has already read, and refuses it before it leaves the cluster. `--openappa` deploys it and sets these flags; see [OpenAPPA](../guides/openappa.md). - **Ingress:** planned. ## Replace: bring your own gateway diff --git a/manifests/ate-install/components/cordon-system-infra-components/kustomization.yaml b/manifests/ate-install/components/cordon-system-infra-components/kustomization.yaml index 847c0cf875..4fb79c0ed9 100644 --- a/manifests/ate-install/components/cordon-system-infra-components/kustomization.yaml +++ b/manifests/ate-install/components/cordon-system-infra-components/kustomization.yaml @@ -32,8 +32,9 @@ # Name-regex targets rather than one patch per workload: the install applies # these workloads through several different streams (the base bundle, a lone # ate-api-server redeploy, the podcert overlay, the postgres file, the egress -# variants, and the bundled credential provider), and kustomize leaves a stream -# alone when nothing in it matches, so the same component fits every path. +# variants, the bundled credential provider, and OpenAPPA), and kustomize +# leaves a stream alone when nothing in it matches, so the same component fits +# every path. apiVersion: kustomize.config.k8s.io/v1alpha1 kind: Component @@ -41,7 +42,7 @@ kind: Component patches: - target: kind: Deployment - name: ate-api-server|ate-idp-server|ate-controller|atenet-router|atenet-egress|podcertificate-controller|k8s-credential-provider|fake-workersync + name: ate-api-server|ate-idp-server|ate-controller|atenet-router|atenet-egress|podcertificate-controller|k8s-credential-provider|fake-workersync|openappa patch: |- apiVersion: apps/v1 kind: Deployment diff --git a/manifests/openappa/appa.toml b/manifests/openappa/appa.toml new file mode 100644 index 0000000000..d9f7a68164 --- /dev/null +++ b/manifests/openappa/appa.toml @@ -0,0 +1,26 @@ +# Copyright 2026 The Agent Substrate Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The policy --openappa installs when the cluster has none. Includes the +# GitHub battery: a repository read labels the session with who may read that +# repository, and a write may carry only what the target's readers may +# already see. +include = ["batteries/github/appa.toml"] + +[policy] +version = 2 + +[externals] +timeout_ms = 10000 +max_body_bytes = 1048576 diff --git a/manifests/openappa/openappa.yaml b/manifests/openappa/openappa.yaml new file mode 100644 index 0000000000..9c57d480e7 --- /dev/null +++ b/manifests/openappa/openappa.yaml @@ -0,0 +1,174 @@ +# Copyright 2026 The Agent Substrate Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# OpenAPPA, the egress gateway's additional ext_proc filter. ate-setup applies +# this under --openappa: it sets both images and appends one --mcp-server flag +# per --openappa-mcp-servers entry. The appa.toml policy is the +# openappa-policy ConfigMap, which ate-setup applies on its own. See +# docs/guides/openappa.md. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: openappa + namespace: ate-system +automountServiceAccountToken: false +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: openappa + namespace: ate-system + labels: + app: openappa +spec: + replicas: 1 + # The runtime keeps its session state in an emptyDir, so two pods would + # each see half of an actor's reads. + strategy: + type: Recreate + selector: + matchLabels: + app: openappa + template: + metadata: + labels: + app: openappa + spec: + serviceAccountName: openappa + automountServiceAccountToken: false + securityContext: + runAsUser: 65532 + runAsGroup: 65532 + runAsNonRoot: true + containers: + - name: extproc + image: ${OPENAPPA_EXTPROC_IMAGE} + args: + - --listen=:9443 + - --admin-listen=127.0.0.1:8081 + - --runtime-url=http://127.0.0.1:8787 + # Serve with the pod's servicedns identity (SAN openappa.ate-system.svc) + # and accept only the egress gateway's podidentity client certificate. + - --tls-credential-bundle=/run/servicedns/credential-bundle.pem + - --tls-client-ca=/run/podidentity-ca/trust-bundle.pem + - --tls-client-id=spiffe://cluster.local/ns/ate-system/sa/atenet-egress + ports: + - name: grpc + containerPort: 9443 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: servicedns + mountPath: /run/servicedns + readOnly: true + - name: podidentity-ca + mountPath: /run/podidentity-ca + readOnly: true + - name: runtime + image: ${OPENAPPA_RUNTIME_IMAGE} + command: ["appa"] + args: + - runtime + - --adapter=kagent + - --listen=127.0.0.1:8787 + - --config=/etc/appa/appa.toml + - --db=/var/lib/appa/appa.db + env: + - name: APPA_BATTERY_TARGET + value: single-user + # Provider tokens, such as APPA_PROVIDER_GITHUB_TOKEN. The operator + # creates the Secret; the runtime starts without it. + envFrom: + - secretRef: + name: openappa-provider-credentials + optional: true + readinessProbe: + exec: + command: ["curl", "-fsS", "http://127.0.0.1:8787/health"] + periodSeconds: 10 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: policy + mountPath: /etc/appa + readOnly: true + - name: state + mountPath: /var/lib/appa + volumes: + - name: policy + configMap: + name: openappa-policy + - name: state + emptyDir: {} + - name: servicedns + projected: + sources: + - podCertificate: + signerName: servicedns.podcert.ate.dev/identity + keyType: ECDSAP256 + credentialBundlePath: credential-bundle.pem + - name: podidentity-ca + projected: + sources: + - clusterTrustBundle: + signerName: podidentity.podcert.ate.dev/identity + labelSelector: + matchLabels: + podcert.ate.dev/canarying: live + path: trust-bundle.pem +--- +apiVersion: v1 +kind: Service +metadata: + name: openappa + namespace: ate-system +spec: + type: ClusterIP + selector: + app: openappa + ports: + - name: grpc + port: 9443 + targetPort: grpc + protocol: TCP +--- +# Only the egress gateway may reach the ext_proc port, and the processor also +# pins the client certificate to the gateway's identity. The runtime and the +# admin listener are on loopback. Takes effect only on a cluster whose network +# plugin enforces NetworkPolicy. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: openappa + namespace: ate-system +spec: + podSelector: + matchLabels: + app: openappa + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: atenet-egress + ports: + - protocol: TCP + port: 9443