Skip to content

Add otel-collector - #493

Open
jorbaum wants to merge 8 commits into
cloudfoundry:mainfrom
jorbaum:add-otel-collector
Open

jorbaum wants to merge 8 commits into
cloudfoundry:mainfrom
jorbaum:add-otel-collector

Conversation

@jorbaum

@jorbaum jorbaum commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds the otel-collector to kind deployment as its commonly used in cloudfoundry already. Deployment of otel-collector is opt-in for now.

Requires cloudfoundry/cf-k8s-releases#230 (CI currently failing due to that).

I verified locally that I am able to send otlp traffic to otelCollector.exportEndpoint via:

$ docker run --detach --name opencode-otel-collector-test --publish 4318:4317 --volume "$PWD/docs/otel-collector-local-config.yaml:/etc/otel/config.yaml:ro" otel/opentelemetry-collector-k8s:0.127.0 --config=/etc/otel/config.yaml
  external_host=$(docker exec cfk8s-control-plane sh -c "ip route | awk '/default/ {print \$3; exit}'")
  cat > /private/var/folders/gl/jsptwkgj1zq1j9xjslk0014h0000gn/T/opencode/otel-external-runtime.yaml <<EOF
  otelCollector:
    enabled: true
    exporter:
      endpoint: "${external_host}:4318"
  charts:
    loggregatorAgent:
      url: /Users/myuser/Workspace/public/cloudfoundry/cf-k8s-releases/loggregator-agent/helm
  EOF
  printf 'external_endpoint=%s:4318\n' "$external_host"
  for signal in ResourceLogs ResourceMetrics; do
    count=$(docker logs --since 5m opencode-otel-collector-test 2>&1 | grep -c "$signal" || true)
    echo "$signal=$count"
  done

Unable to find image 'otel/opentelemetry-collector-k8s:0.127.0' locally
0.127.0: Pulling from otel/opentelemetry-collector-k8s
ccb850397780: Pulling fs layer
1cb5be31027a: Pulling fs layer
1cb5be31027a: Download complete
1cb5be31027a: Pull complete
ccb850397780: Download complete
ccb850397780: Pull complete
Digest: sha256:96cacfd787592737d68d239451ded1661a27cdc106e7c4413af287ca60940e1e
Status: Downloaded newer image for otel/opentelemetry-collector-k8s:0.127.0
ff9abd377266f517404278997eefb512118c23ffb5009390298257e7584b42bd
ResourceLogs=0
ResourceMetrics=32

and internal otel-collector:

❯ kubectl --context kind-cfk8s -n cf-system logs daemonset/forwarder-agent \
  -c otel-collector --all-pods=true --prefix=true --since=1m -f |
  awk '/ResourceMetrics|Metric #[0-9]+/{print; fflush()}' | head
[pod/forwarder-agent-7wwhc/otel-collector] 2026-10-20T15:21:10.451Z	info	ResourceMetrics #0
[pod/forwarder-agent-7wwhc/otel-collector] Metric #0
[pod/forwarder-agent-7wwhc/otel-collector] 2026-10-20T15:21:10.652Z	info	ResourceMetrics #0
[pod/forwarder-agent-7wwhc/otel-collector] Metric #0
[pod/forwarder-agent-7wwhc/otel-collector] Metric #1
[pod/forwarder-agent-7wwhc/otel-collector] Metric #2
[pod/forwarder-agent-7wwhc/otel-collector] Metric #3
[pod/forwarder-agent-7wwhc/otel-collector] Metric #4
[pod/forwarder-agent-7wwhc/otel-collector] 2026-10-20T15:21:10.853Z	info	ResourceMetrics #0
[pod/forwarder-agent-7wwhc/otel-collector] Metric #0

Drafted by an AI coding harness, carefully reviewed.

Comment thread helmfile.yaml.gotmpl Outdated
Comment thread versions.yaml Outdated
@jorbaum

jorbaum commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

With this next try I am now using the upstream image from otel/opentelemetry-collector-contrib .

However, I also adapted the loggregator-agent release helm chart so it includes the otel-collector. This is necessary as the forwarder-agent currently only forwards to localhost.

Alternatively one could adapt the forwarder agent to send OTLP to other locations as well. However, when we think about going that route we could also add a proper cloudfoundry loggregator envelope receiver to the otel-collector as well. I consider this out of scope for now.

@jorbaum jorbaum changed the title Add otel-collector-release based on upstream helm Add otel-collector-release Sep 25, 2026
@jorbaum jorbaum changed the title Add otel-collector-release Add otel-collector Sep 25, 2026
@jorbaum

jorbaum commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

I verified locally that I am able to send otlp traffic to otelCollector.exportEndpoint.

Does it make sense to add a test for that?

jorbaum and others added 5 commits October 13, 2026 16:59
Uses upstream helm chart but our own otel distro
* Also configures healthcheck for otel collector
otel-collector is still experimental and should not be deployed by
default
Re-use the upstream otel/opentelemetry-collector-contrib image instead of
the CF-maintained distro. Drops the cross-repo deps on cf-k8s-releases#230
and otel-collector-release#263 — the contrib image already bundles
health_check.

- Replace CF distro image + custom command with otel/opentelemetry-collector-contrib
- Add mTLS OTLP receiver (otlp/cf-internal-local, grpc 0.0.0.0:4317, TLS 1.3)
  mirroring the BOSH otel-collector-release job, using all-in-one-tls secret
- Mount all-in-one-tls at /etc/otel/certs (extraVolumes/extraVolumeMounts)
- Add otlp exporter targeting configurable exportEndpoint (default: host.docker.internal:4317)
- Document config lineage in comment pointing to upstream chart values.yaml
- Drop imageTag from versions.yaml; add exportEndpoint value

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
forwarder-agent only sends to localhost right now
Moved enabled and exportEndpoint under otelCollector in values.yaml.gotmpl
@jorbaum
jorbaum force-pushed the add-otel-collector branch from 712d8b7 to aac58b4 Compare October 13, 2026 15:49
@jorbaum
jorbaum force-pushed the add-otel-collector branch from aac58b4 to 31c28b4 Compare October 20, 2026 15:16
@jorbaum

jorbaum commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Implemented what we had discussed in cloudfoundry/cf-k8s-releases#230 (comment) .

Also moved the otel config into dedicated template file to follow usual conventions.

@jorbaum
jorbaum requested a review from pbusko October 20, 2026 15:32
@jorbaum
jorbaum marked this pull request as ready for review October 20, 2026 15:32
@jorbaum
jorbaum requested a review from a team as a code owner October 20, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants