Skip to content

The Kubernetes credential provider exports metrics to the cluster collector - #3379

Open
Eric Curtin (ericcurtin) wants to merge 5 commits into
agent-substrate:mainfrom
ericcurtin:credential-provider-otel-config
Open

Eric Curtin (ericcurtin) wants to merge 5 commits into
agent-substrate:mainfrom
ericcurtin:credential-provider-otel-config

Conversation

@ericcurtin

Copy link
Copy Markdown
Contributor

Without envFrom for ate-otel-config the provider's OTLP exporter dialed localhost:4317, so its metrics were lost. Adds it, plus the pod resource attributes the other components set.

Fixes #3282

AI use: AI-assisted (opencode). I reviewed the diff and ran the tests.

  • Tests pass
  • Docs updated

…lector

The Deployment had no envFrom for ate-otel-config, so the OTLP exporter dialed localhost:4317. It now also sets the pod resource attributes like the other components.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Lior Lieberman (@LiorLieberman) Haven Xia (@HavenXia) PTAL when you get a chance. Thank you!

@ericcurtin

Copy link
Copy Markdown
Contributor Author

Julian Gutierrez Oschmann (@juli4n) Jeff Luo (@JeffLuoo) PTAL when you get a chance. CI is green. Thank you!

@JeffLuoo Jeff Luo (JeffLuoo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. The envFrom change is correct. I found two gaps outside the diff that I think must be fixed before merge. Both come from the same cause: the provider is now a consumer of ate-otel-config, but the restart code and the docs do not include it.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Done: provider added to otelOverrideDeployments with a test case, README and docs/observability.md updated, and the ConfigMap comment says it comes from manifests/egress-credential-injection. Thanks!

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Jeff Luo (@JeffLuoo) Haven Xia (@HavenXia) PTAL when you get a chance, feedback addressed. Thank you!

@JeffLuoo Jeff Luo (JeffLuoo) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, both blockers are fixed. I approve this PR. Because of the release freeze, it can merge on 11/18 or later.

Three small changes before the merge, if you agree:

  1. cmd/ate-setup/internal/steps/overlay.go: use the k8sCredentialProviderDeployment constant instead of the string "k8s-credential-provider", here and in overlay_test.go. TestK8sCredentialProviderManifestsAgree compares that constant with the manifest. If someone renames the Deployment, the restart stops with no error, because RolloutRestartDeployment ignores a missing workload.
  2. docs/guides/observability.md: the provider calls only serverboot.InitMetrics, so it sends no traces and no OTLP logs. Please remove it from the list of components whose telemetry goes to Jaeger and to the debug exporter, and add a sentence that says that it sends only metrics.
  3. manifests/ate-install/ate-otel-config.yaml: "so restart it after editing this file" makes it look as if the other consumers restart automatically. They do not (see the NOTE below it). Suggestion: "...comes from manifests/egress-credential-injection, so a restart of the bundle workloads does not restart it."

…y metrics

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Jeff Luo (@JeffLuoo) Done in ffcf842, thank you!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

k8s-credential-provider manifest is missing ate-otel-config envFrom, failing OTLP metric exports to [::1]:4317 / 127.0.0.1:4317 every 60s

2 participants