Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions cmd/ate-setup/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ every setting with its environment variable.
| `--ingress-allowed-spiffe-ids IDS` | Comma-separated SPIFFE IDs (exact match) of the clients the ingress router lets reach actors; any listed client may reach any actor. Unset, no client may reach actors through the router. Both dataplanes require a Pod Identity CA client certificate on HTTPS and TLS CONNECT. The SPIFFE URI SAN allowlist may be checked during or after TLS, including after CONNECT acceptance, but unlisted clients must be denied before actor routing. The podidentity signer issues `spiffe://cluster.local/ns/<namespace>/sa/<service-account>`. `ATE_INGRESS_ALLOWED_SPIFFE_IDS` when the flag is absent |
| `--egress-gateway-extra-spiffe-ids IDS` | Comma-separated SPIFFE IDs that hold ate-api-server's `egress_gateway` role next to atenet-egress, such as a credential provider that mints actor JWTs. ate-setup passes atenet-egress's ID and then each listed ID as repeated `--egress-gateway-spiffe-id` flags; unset passes none, leaving ate-api-server's default. `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` when the flag is absent |
| `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter |
| `--experimental-additional-egress-extproc-allow-mode-override` | Let that processor buffer the request and response bodies of a request it picks, through a `mode_override` in its `request_headers` response. See [`docs/security/extensibility.md`](../../docs/security/extensibility.md). Requires `--experimental-additional-egress-extproc-service`. `ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE=true` when the flag is absent |
| `--experimental-additional-egress-extproc-timeout DURATION` | How long Envoy waits for each response from that processor (default `2s`, at most `1h`). Requires `--experimental-additional-egress-extproc-service`. `ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT` when the flag is absent |
| `--openappa` | Deploy [OpenAPPA](../../docs/guides/openappa.md) as the egress gateway's additional ext_proc filter, on `deploy ate-system` and `deploy atenet`. It sets the three `--experimental-additional-egress-extproc-*` flags above, refusing another service or a mode override turned off, and a redeploy without it removes OpenAPPA. Requires `--atenet-dataplane=envoy`. `ATE_OPENAPPA=true` when the flag is absent |
| `--openappa-policy PATH` | appa.toml to apply on every deploy. Unset, the bundled policy is applied only when the cluster has none. `ATE_OPENAPPA_POLICY` when the flag is absent |
| `--openappa-mcp-servers HOST=NAME,...` | MCP servers OpenAPPA checks tool calls to (default `api.githubcopilot.com=github`). `ATE_OPENAPPA_MCP_SERVERS` when the flag is absent |
| `--openappa-runtime-image IMAGE`, `--openappa-extproc-image IMAGE` | Override OpenAPPA's images. `ATE_OPENAPPA_RUNTIME_IMAGE` and `ATE_OPENAPPA_EXTPROC_IMAGE` when the flags are absent |
| `--credential-provider JSON` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"<provider>","address":"<host>:<port>"}` uses a provider you deploy yourself. A JSON array of provider objects selects several, each serving the `ate-secret://` URIs its name is the host of. See [`docs/guides/egress-credential-injection.md`](../../docs/guides/egress-credential-injection.md). `ATE_CREDENTIAL_PROVIDER` when the flag is absent |
| `--ateapi-log-level LEVEL` | ate-api-server log level: `debug`, `info`, `warn`, or `error`. Written to the `ate-api-config` ConfigMap on every deploy when set; unset keeps the installed level (`info` on a new install). Changing the level updates the env-hash annotation and rolls `ate-api-server`. `ATE_API_LOG_LEVEL` when the flag is absent |
| `--otlp-endpoint URL` | Send control plane telemetry to `URL` instead of the cluster default (see [`benchmarking/telemetry/README.md`](../../benchmarking/telemetry/README.md)). `ATE_OTLP_ENDPOINT` when the flag is absent |
Expand Down
40 changes: 40 additions & 0 deletions cmd/ate-setup/internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,25 @@ type Config struct {
// AdditionalEgressExtprocService is the optional NS/SVC:PORT external processor filter.
AdditionalEgressExtprocService string

// AdditionalEgressExtprocAllowModeOverride lets that processor switch a
// request to buffered body processing with a mode_override.
AdditionalEgressExtprocAllowModeOverride bool

// AdditionalEgressExtprocTimeout is that filter's message_timeout and gRPC
// timeout.
AdditionalEgressExtprocTimeout time.Duration

// OpenAPPA deploys OpenAPPA as the additional ext_proc filter
// (ATE_OPENAPPA). It implies the three settings above.
OpenAPPA bool
// OpenAPPAPolicy is a local appa.toml. Empty means the bundled policy,
// applied only where the cluster has none.
OpenAPPAPolicy string
// OpenAPPAMCPServers are the HOST=NAME pairs OpenAPPA checks tool calls to.
OpenAPPAMCPServers []string
OpenAPPARuntimeImage string
OpenAPPAExtprocImage string

// CredentialProviderJSON is the --credential-provider value; see
// CredentialProviders for its schema.
CredentialProviderJSON string
Expand Down Expand Up @@ -532,6 +551,15 @@ func (c *Config) ScriptEnv() []string {
}
if c.AdditionalEgressExtprocService != "" {
merged["ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"] = c.AdditionalEgressExtprocService
merged["ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT"] = c.AdditionalEgressExtprocTimeout.String()
}
delete(merged, "ATE_OPENAPPA")
if c.OpenAPPA {
merged["ATE_OPENAPPA"] = "true"
}
delete(merged, "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE")
if c.AdditionalEgressExtprocAllowModeOverride {
merged["ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE"] = "true"
}
if c.CredentialProviderJSON != "" {
merged["ATE_CREDENTIAL_PROVIDER"] = c.CredentialProviderJSON
Expand Down Expand Up @@ -576,6 +604,18 @@ const (
K8sCredentialProviderAddress = "k8s-credential-provider." + installdefaults.SystemNamespace + ".svc:50051"
)

// OpenAPPA's Service in manifests/openappa, which the egress gateway reaches
// as its additional ext_proc filter, and its default images.
const (
OpenAPPAExtprocService = installdefaults.SystemNamespace + "/openappa:9443"
OpenAPPARuntimeImage = "ghcr.io/archestra-ai/appa-runtime:v0.42.0@sha256:4cc8f3f685a725106aba5ea05f022447799d11a70a34a0123c96bd73ff27b5ca"
OpenAPPAExtprocImage = "ghcr.io/archestra-ai/appa-extproc:v0.43.0"

// openAPPAExtprocTimeout leaves room for the policy's external calls,
// such as the GitHub battery's.
openAPPAExtprocTimeout = "60s"
)

// credentialProviderKey is the setting the provider JSON arrives on.
const credentialProviderKey = "atenet.egress.credentialProvider"

Expand Down
116 changes: 116 additions & 0 deletions cmd/ate-setup/internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ func loadEnv(t *testing.T) {
"ACTOR_JWT_ALGORITHM",
"ACTOR_JWT_ISSUER",
"ANTHROPIC_API_KEY",
"ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE",
"ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE",
"ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT",
"ATE_API_AUTHZ_BOOTSTRAP_OWNERS",
"ATE_API_LOG_LEVEL",
"ATE_API_POSTGRES_CLOUDSQL_GSA",
Expand All @@ -72,6 +74,11 @@ func loadEnv(t *testing.T) {
"ATE_INSTALL_KIND",
"ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER",
"ATE_INSTALL_ROLLOUT_TIMEOUT",
"ATE_OPENAPPA",
"ATE_OPENAPPA_EXTPROC_IMAGE",
"ATE_OPENAPPA_MCP_SERVERS",
"ATE_OPENAPPA_POLICY",
"ATE_OPENAPPA_RUNTIME_IMAGE",
"ATE_OTLP_ENDPOINT",
"ATE_PREVIEW",
"BENCHMARK_ACTOR_MEMORY",
Expand Down Expand Up @@ -278,6 +285,101 @@ func TestLoadE2ETuning(t *testing.T) {
}
}

func TestLoadAdditionalEgressExtproc(t *testing.T) {
const service = "ate-system/extproc:50051"
for _, tc := range []struct {
name string
opts Options
env map[string]string
wantOverride bool
wantTimeout time.Duration
}{
{name: "defaults", opts: Options{AdditionalEgressExtprocService: service}, wantTimeout: 2 * time.Second},
{
name: "flags",
opts: Options{
AdditionalEgressExtprocService: service,
AdditionalEgressExtprocAllowModeOverride: true,
AdditionalEgressExtprocTimeout: "10s",
},
wantOverride: true, wantTimeout: 10 * time.Second,
},
{
name: "environment",
opts: Options{AdditionalEgressExtprocService: service},
env: map[string]string{
"ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "true",
"ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "500ms",
},
wantOverride: true, wantTimeout: 500 * time.Millisecond,
},
} {
t.Run(tc.name, func(t *testing.T) {
loadEnv(t)
for k, v := range tc.env {
t.Setenv(k, v)
}
cfg, err := Load(tc.opts)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.AdditionalEgressExtprocAllowModeOverride != tc.wantOverride {
t.Errorf("AdditionalEgressExtprocAllowModeOverride = %v, want %v", cfg.AdditionalEgressExtprocAllowModeOverride, tc.wantOverride)
}
if cfg.AdditionalEgressExtprocTimeout != tc.wantTimeout {
t.Errorf("AdditionalEgressExtprocTimeout = %v, want %v", cfg.AdditionalEgressExtprocTimeout, tc.wantTimeout)
}
if _, exported := scriptEnvMap(t, cfg)["ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE"]; exported != tc.wantOverride {
t.Errorf("ScriptEnv() exports ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE = %v, want %v", exported, tc.wantOverride)
}
})
}
}

// --openappa points the additional ext_proc filter at OpenAPPA, keeping any
// timeout a channel supplied.
func TestLoadOpenAPPA(t *testing.T) {
for _, tc := range []struct {
name string
opts Options
wantTimeout time.Duration
}{
{name: "implied", opts: Options{OpenAPPA: true}, wantTimeout: 60 * time.Second},
{name: "own timeout", opts: Options{OpenAPPA: true, AdditionalEgressExtprocTimeout: "5s"}, wantTimeout: 5 * time.Second},
{
name: "same service named",
opts: Options{OpenAPPA: true, AdditionalEgressExtprocService: OpenAPPAExtprocService},
wantTimeout: 60 * time.Second,
},
} {
t.Run(tc.name, func(t *testing.T) {
loadEnv(t)
cfg, err := Load(tc.opts)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if cfg.AdditionalEgressExtprocService != OpenAPPAExtprocService {
t.Errorf("AdditionalEgressExtprocService = %q, want %q", cfg.AdditionalEgressExtprocService, OpenAPPAExtprocService)
}
if !cfg.AdditionalEgressExtprocAllowModeOverride {
t.Error("AdditionalEgressExtprocAllowModeOverride = false, want true")
}
if cfg.AdditionalEgressExtprocTimeout != tc.wantTimeout {
t.Errorf("AdditionalEgressExtprocTimeout = %v, want %v", cfg.AdditionalEgressExtprocTimeout, tc.wantTimeout)
}
if want := []string{"api.githubcopilot.com=github"}; !slices.Equal(cfg.OpenAPPAMCPServers, want) {
t.Errorf("OpenAPPAMCPServers = %v, want %v", cfg.OpenAPPAMCPServers, want)
}
if cfg.OpenAPPARuntimeImage != OpenAPPARuntimeImage || cfg.OpenAPPAExtprocImage != OpenAPPAExtprocImage {
t.Errorf("images = %q, %q, want the defaults", cfg.OpenAPPARuntimeImage, cfg.OpenAPPAExtprocImage)
}
if got := scriptEnvMap(t, cfg)["ATE_OPENAPPA"]; got != "true" {
t.Errorf("ScriptEnv() ATE_OPENAPPA = %q, want true", got)
}
})
}
}

func TestLoadIngressAllowedSPIFFEIDs(t *testing.T) {
const a, b = "spiffe://cluster.local/ns/a/sa/a", "spiffe://cluster.local/ns/b/sa/b"
for _, tc := range []struct {
Expand Down Expand Up @@ -1042,6 +1144,20 @@ func TestLoadRejectsInvalidValues(t *testing.T) {
{"cluster size", Options{ClusterSize: "size5"}},
{"extproc invalid format", Options{AdditionalEgressExtprocService: "extproc:50051"}},
{"extproc agentgateway", Options{Router: RouterAgentgateway, AdditionalEgressExtprocService: "ate-system/extproc:50051"}},
{"extproc mode override without a service", Options{AdditionalEgressExtprocAllowModeOverride: true}},
{"extproc timeout without a service", Options{AdditionalEgressExtprocTimeout: "5s"}},
{"extproc timeout zero", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "0s"}},
{"extproc timeout negative", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "-1s"}},
{"extproc timeout over an hour", Options{AdditionalEgressExtprocService: "ate-system/extproc:50051", AdditionalEgressExtprocTimeout: "61m"}},
{"openappa on agentgateway", Options{Router: RouterAgentgateway, OpenAPPA: true}},
{"openappa with another extproc service", Options{OpenAPPA: true, AdditionalEgressExtprocService: "ate-system/extproc:50051"}},
{"openappa policy that does not exist", Options{OpenAPPA: true, OpenAPPAPolicy: "/nonexistent/appa.toml"}},
{"openappa policy is a directory", Options{OpenAPPA: true, OpenAPPAPolicy: "."}},
{"openappa mcp server without a name", Options{OpenAPPA: true, OpenAPPAMCPServers: "api.githubcopilot.com"}},
{"openappa mcp server with an empty host", Options{OpenAPPA: true, OpenAPPAMCPServers: "=github"}},
{"openappa mcp server as a URL", Options{OpenAPPA: true, OpenAPPAMCPServers: "https://api.githubcopilot.com=github"}},
{"openappa policy without openappa", Options{OpenAPPAPolicy: "appa.toml"}},
{"openappa mcp servers without openappa", Options{OpenAPPAMCPServers: "a.example.com=a"}},
{"provider not JSON", Options{CredentialProvider: "k8s.io"}},
{"provider as a JSON string", Options{CredentialProvider: `"off"`}},
{"provider null", Options{CredentialProvider: `null`}},
Expand Down
18 changes: 17 additions & 1 deletion cmd/ate-setup/internal/config/coverage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,13 @@
package config

import (
"os"
"path/filepath"
"reflect"
"strconv"
"testing"

"go.ate.dev/substrate/internal/installdefaults"
)

// notFromASetting are Config fields no setting feeds: derived values, the
Expand Down Expand Up @@ -58,6 +62,12 @@ func TestEveryConfigFieldIsFedBySomeSetting(t *testing.T) {
}
}

policy := filepath.Join(t.TempDir(), "appa.toml")
if err := os.WriteFile(policy, nil, 0o600); err != nil {
t.Fatal(err)
}
env["ATE_OPENAPPA_POLICY"] = policy

r, err := Resolve(nil, ResolveOptions{Env: env})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
Expand Down Expand Up @@ -95,7 +105,13 @@ func valueFor(key string) string {
case "ateapi.postgres.cloudsql.ipType":
return CloudSQLIPTypePSC
case "atenet.egress.additionalExtprocService":
return "ns/svc:8080"
// openappa.enabled is set too, and refuses any other service.
return OpenAPPAExtprocService
case "openappa.mcpServers":
return "mcp.example.com=example"
case "namespace":
// openappa.enabled is set too, and its manifest names this namespace.
return installdefaults.SystemNamespace
case "atenet.egress.credentialProvider":
return `{"name":"k8s.io"}`
case "actorJWT.algorithm":
Expand Down
44 changes: 44 additions & 0 deletions cmd/ate-setup/internal/config/errors_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,50 @@ func TestValidationErrorsNameTheSupplyingChannel(t *testing.T) {
env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:99999"},
wantIn: []string{"atenet.egress.additionalExtprocService", "1-65535", "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"},
},
{
name: "extproc mode override without a service",
env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "true"},
wantIn: []string{"atenet.egress.additionalExtprocAllowModeOverride", "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", "atenet.egress.additionalExtprocService"},
},
{
name: "extproc timeout without a service",
env: map[string]string{"ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "5s"},
wantIn: []string{"atenet.egress.additionalExtprocTimeout", "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT", "atenet.egress.additionalExtprocService"},
},
{
name: "extproc timeout is zero",
env: map[string]string{
"ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:50051",
"ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT": "0s",
},
wantIn: []string{"atenet.egress.additionalExtprocTimeout", "at most 1h", "ATE_ADDITIONAL_EGRESS_EXTPROC_TIMEOUT"},
},
{
name: "openappa with another extproc service",
env: map[string]string{
"ATE_OPENAPPA": "true",
"ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE": "ns/svc:50051",
},
wantIn: []string{"atenet.egress.additionalExtprocService", "ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE", "openappa.enabled", "ATE_OPENAPPA", OpenAPPAExtprocService},
},
{
name: "openappa with mode overrides turned off",
env: map[string]string{
"ATE_OPENAPPA": "true",
"ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE": "false",
},
wantIn: []string{"atenet.egress.additionalExtprocAllowModeOverride", "ATE_ADDITIONAL_EGRESS_EXTPROC_ALLOW_MODE_OVERRIDE", "openappa.enabled"},
},
{
name: "openappa in another namespace",
env: map[string]string{"ATE_OPENAPPA": "true", "ATE_NAMESPACE": "elsewhere"},
wantIn: []string{"openappa.enabled", "namespace", "ATE_NAMESPACE", "manifests/openappa"},
},
{
name: "openappa image without openappa",
env: map[string]string{"ATE_OPENAPPA_RUNTIME_IMAGE": "example.com/runtime:v1"},
wantIn: []string{"openappa.runtimeImage", "ATE_OPENAPPA_RUNTIME_IMAGE", "openappa.enabled"},
},
{
name: "image tag without a repository",
env: map[string]string{"ATE_IMAGE_TAG": "v1"},
Expand Down
Loading