Skip to content

Replace OpenCensus with OpenTelemetry for metrics - #1

Draft
MartinForReal with Copilot wants to merge 186 commits into
masterfrom
copilot/replace-opensensus-with-opentelemetry
Draft

MartinForReal with Copilot wants to merge 186 commits into
masterfrom
copilot/replace-opensensus-with-opentelemetry

Conversation

Copilot AI commented Dec 25, 2025 •

Copy link
Copy Markdown

What type of PR is this?

/kind feature

What this PR does / why we need it:

Replaces OpenCensus with OpenTelemetry for metrics instrumentation. OpenCensus has been deprecated and merged into OpenTelemetry, which is now the CNCF standard for observability.

Which issue(s) this PR fixes:

Fixes the need to migrate from deprecated OpenCensus to OpenTelemetry.

Special notes for your reviewer:

Changes

Metrics Package (pkg/util/metrics/)

  • Replace go.opencensus.io/tag → go.opentelemetry.io/otel/attribute
  • Replace go.opencensus.io/stats → go.opentelemetry.io/otel/metric SDK
  • Add provider.go for global MeterProvider management
  • Fix ParsePrometheusMetrics to skip unsupported metric types (SUMMARY, HISTOGRAM) instead of erroring

Exporters

  • Prometheus: Use go.opentelemetry.io/otel/exporters/prometheus instead of OpenCensus contrib exporter
    • Use WithTranslationStrategy(otlptranslator.UnderscoreEscapingWithoutSuffixes) to prevent adding _ratio or _total suffixes
    • Add WithoutScopeInfo() to prevent adding otel_scope_* labels
  • Stackdriver: Use github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric instead of OpenCensus contrib exporter

Initialization Order Fix

  • Exporters are now set up before SetupMeterProvider() is called
  • problemmetrics.GlobalProblemMetricsManager uses lazy initialization to defer metric creation
  • This ensures metrics are created after the meter provider is configured with all readers

Dependencies

  • Remove: go.opencensus.io, contrib.go.opencensus.io/*
  • Add: go.opentelemetry.io/otel/sdk/metric, go.opentelemetry.io/otel/exporters/prometheus
  • Move github.com/prometheus/otlptranslator to direct dependency (used for WithTranslationStrategy)

API Unchanged

The wrapper API remains identical:

// Same API, different implementation
metric, _ := metrics.NewInt64Metric(id, "name", "desc", "unit", metrics.Sum, []string{"tag1"})
metric.Record(map[string]string{"tag1": "value"}, 42)

Internally, LastValue aggregation maps to OTel gauges, Sum maps to OTel counters.

Does this PR introduce a user-facing change?

Replace OpenCensus with OpenTelemetry for metrics collection and export. No changes to the metrics API.
Original prompt

replace opensensus with opentelemetry and use wrapper in code repo, no new wrapper added. make sure the behavior is the same after the migration


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

arjraman and others added 2 commits December 11, 2025 15:51
Removed log message for disabled restart on error.
Copilot AI changed the title [WIP] Replace OpenCensus with OpenTelemetry and implement wrapper Replace OpenCensus with OpenTelemetry for metrics Dec 25, 2025
Copilot AI requested a review from MartinForReal December 25, 2025 03:46
@MartinForReal

Copy link
Copy Markdown
Owner

@copilot update the description kubernetes#1210

Copilot AI commented Dec 25, 2025

Copy link
Copy Markdown
Author

@copilot update the description kubernetes#1210

Updated the PR description to match the kubernetes/node-problem-detector PR template format, including the /kind feature tag, structured sections, and release notes.

@MartinForReal

Copy link
Copy Markdown
Owner

@copilot no. update the description of pr kubernetes#1210 this pr will be contributed to remote repo.

Copilot AI commented Dec 25, 2025

Copy link
Copy Markdown
Author

@copilot no. update the description of pr kubernetes#1210 this pr will be contributed to remote repo.

I don't have the capability to update PR descriptions on external repositories like kubernetes/node-problem-detector. I can only update the description of this fork's PR. You'll need to manually update PR kubernetes#1210's description on the upstream repository, or you can copy the description I've prepared here to use there.

@MartinForReal

Copy link
Copy Markdown
Owner

@copilot build the project and validate the metrics format

Copilot AI commented Dec 25, 2025

Copy link
Copy Markdown
Author

@copilot build the project and validate the metrics format

Build and metrics validation completed successfully:

Build: ✅ go build ./... passes

Tests: ✅ All tests pass

Metrics Format Validation: ✅ Prometheus metrics exported correctly:

# HELP test_counter_total A test counter metric
# TYPE test_counter_total counter
test_counter_total{reason="TestReason1",...} 5
test_counter_total{reason="TestReason2",...} 10

# HELP test_gauge_seconds A test gauge metric
# TYPE test_gauge_seconds gauge
test_gauge_seconds{instance="node1",...} 12345

OpenTelemetry metadata is properly included (otel_scope_name="k8s.io/node-problem-detector", telemetry_sdk_name="opentelemetry", version 1.38.0).

@MartinForReal

Copy link
Copy Markdown
Owner

@copilot [2] NPD should export Prometheus metrics.
[2] /home/prow/go/src/k8s.io/node-problem-detector/test/e2e/metriconly/metrics_test.go:36
[2] On a clean node
[2] /home/prow/go/src/k8s.io/node-problem-detector/test/e2e/metriconly/metrics_test.go:69
[2] NPD should not report any problem [It]
[2] /home/prow/go/src/k8s.io/node-problem-detector/test/e2e/metriconly/metrics_test.go:101
[2]
[2] Expect NPD to become ready in 120s, but hit error: All attempts fail:
[2] #1: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#2: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#3: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#4: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#5: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#6: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#7: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#8: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#9: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#10: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#11: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds
[2] kubernetes#12: Error fetching NPD metrics: error parsing NPD metrics: unexpected MetricType SUMMARY for metric go_gc_duration_seconds

Copilot AI commented Dec 25, 2025

Copy link
Copy Markdown
Author

@copilot [2] NPD should export Prometheus metrics.
[2] /home/prow/go/src/k8s.io/node-problem-detector/test/e2e/metriconly/metrics_test.go:36
[2] On a clean node...

Fixed in e277f75. The ParsePrometheusMetrics function now skips unsupported metric types (SUMMARY, HISTOGRAM, etc.) instead of returning an error. The OpenTelemetry Prometheus exporter exports Go runtime metrics like go_gc_duration_seconds as SUMMARY type, which NPD doesn't need to parse.

@MartinForReal

Copy link
Copy Markdown
Owner

@copilot push to remote

Copilot AI commented Dec 25, 2025

Copy link
Copy Markdown
Author

@copilot push to remote

I've pushed the fix (4a5e215) to this fork's branch. The issue was that metrics were being created during package init() before the meter provider had the Prometheus exporter attached.

I cannot push directly to the upstream kubernetes/node-problem-detector repository. You'll need to sync this fork's changes to your upstream PR kubernetes#1210 manually.

@MartinForReal

Copy link
Copy Markdown
Owner

@copilot compare with kubernetes#1141 and see if there is anything missing

kubernetes-prow Bot and others added 23 commits September 1, 2026 13:04
…les/k8s-4c11e3f931

chore(deps): bump the k8s group across 2 directories with 4 updates
…actions/actions-all-ebf40551bc

chore(deps): bump the actions-all group with 4 updates
Bumps the actions-all group with 1 update: [step-security/harden-runner](https://github.com/step-security/harden-runner).


Updates `step-security/harden-runner` from 2.21.0 to 2.21.1
- [Release notes](https://github.com/step-security/harden-runner/releases)
- [Commits](step-security/harden-runner@05e3151...e14015d)

---
updated-dependencies:
- dependency-name: step-security/harden-runner
  dependency-version: 2.21.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions-all
...

Signed-off-by: dependabot[bot] <support@github.com>
…actions/actions-all-4c70b186bc

chore(deps): bump step-security/harden-runner from 2.21.0 to 2.21.1 in the actions-all group
Bumps the actions-all group with 4 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.9 to 4.38.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@cdf488f...b96794f)

Updates `github/codeql-action/autobuild` from 4.37.9 to 4.38.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@cdf488f...b96794f)

Updates `github/codeql-action/analyze` from 4.37.9 to 4.38.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@cdf488f...b96794f)

Updates `github/codeql-action/upload-sarif` from 4.37.9 to 4.38.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@cdf488f...b96794f)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.38.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-all
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.38.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-all
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.38.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-all
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.38.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-all
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Max Allan <max.allan@chainguard.dev>
Bumps the k8s group with 3 updates in the / directory: [k8s.io/api](https://github.com/kubernetes/api), [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) and [k8s.io/client-go](https://github.com/kubernetes/client-go).
Bumps the k8s group with 2 updates in the /test directory: [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) and [k8s.io/component-base](https://github.com/kubernetes/component-base).


Updates `k8s.io/api` from 0.37.0 to 0.37.1
- [Commits](kubernetes/api@v0.37.0...v0.37.1)

Updates `k8s.io/apimachinery` from 0.37.0 to 0.37.1
- [Commits](kubernetes/apimachinery@v0.37.0...v0.37.1)

Updates `k8s.io/client-go` from 0.37.0 to 0.37.1
- [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md)
- [Commits](kubernetes/client-go@v0.37.0...v0.37.1)

Updates `k8s.io/apimachinery` from 0.37.0 to 0.37.1
- [Commits](kubernetes/apimachinery@v0.37.0...v0.37.1)

Updates `k8s.io/component-base` from 0.37.0 to 0.37.1
- [Commits](kubernetes/component-base@v0.37.0...v0.37.1)

---
updated-dependencies:
- dependency-name: k8s.io/api
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s
- dependency-name: k8s.io/apimachinery
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s
- dependency-name: k8s.io/client-go
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s
- dependency-name: k8s.io/apimachinery
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s
- dependency-name: k8s.io/component-base
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s
...

Signed-off-by: dependabot[bot] <support@github.com>
The kubekins-e2e master image moved to Go 1.27 on 2026-09-08.
golangci-lint v2.11.4 is built with Go 1.26 and cannot load the Go 1.27
standard library, so `make fmt` (a prerequisite of `make test` and
`make e2e-test`) fails in pull-npd-test, pull-npd-e2e-test and the
periodic ci-npd-test:

  panic: file requires newer Go version go1.27 (application built with
  go1.26)

Switch to v2.14.0, which is built with Go 1.27, and install it from
https://golangci-lint.run/install.sh. The install.sh on the golangci-lint
master branch is no longer maintained and fails checksum verification
for releases >= v2.12.2: its unanchored grep also matches the .sbom.json
entry in the checksums file.

goconst >= 1.9.0 (bundled since golangci-lint v2.12.0) also inspects
composite literals and would report 84 pre-existing string literals, 78
of them in tests. Exclude CompositeLit from goconst so the linting scope
stays the same as before.

The two formatting changes come from running `make fmt` with the new
toolchain (gofmt in Go 1.27 and gofumpt 0.12).
…14.0

Update golangci-lint to v2.14.0 for Go 1.27
…les/k8s-7ec441f29d

chore(deps): bump the k8s group across 2 directories with 4 updates
…actions/actions-all-6b15e67bac

chore(deps): bump the actions-all group with 4 updates
go vet flagged e2e/lib/gce/instance.go: non-constant format string in call to fmt.Errorf. Use errors.New for the pre-formatted message.
Bump Go to 1.27.2 and golang.org/x modules
Fixes CVE-2026-84304 (GO-2026-6348), which is reachable through the gRPC client used by the metric exporters, as well as CVE-2026-84445 (GO-2026-6443) and CVE-2026-84303 (GO-2026-6441).
GO-2026-6505: exporter config logging may leak endpoint URLs. Reachable through the OpenTelemetry meter provider initialization.
Fix Go module CVEs found by scanning the post-submit image and branch head
debian-base is only rebuilt on demand, so run apt-get dist-upgrade on top
of it to pick up current Debian security updates with every image build.
Also install libsystemd0 instead of libsystemd-dev; the runtime image only
needs the shared library.
Upgrade Debian packages when building the image
@MartinForReal
MartinForReal force-pushed the copilot/replace-opensensus-with-opentelemetry branch from 4fbbdf4 to 5e8b123 Compare October 19, 2026 03:54
dependabot Bot and others added 5 commits October 19, 2026 21:33
Bumps build-image/debian-base from bookworm-v1.0.8 to bookworm-v1.0.9.

---
updated-dependencies:
- dependency-name: build-image/debian-base
  dependency-version: bookworm-v1.0.9
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
…build-image/debian-base-bookworm-v1.0.9

chore(deps): bump build-image/debian-base from bookworm-v1.0.8 to bookworm-v1.0.9
Cover a sample line without a # TYPE directive, which the Prometheus text parser treats as an implicitly UNTYPED family, while preserving explicit UNTYPED coverage. Clarify the parser documentation and skip comment so unsupported families are described accurately without changing COUNTER, GAUGE, SUMMARY, or HISTOGRAM handling.
@MartinForReal
MartinForReal force-pushed the copilot/replace-opensensus-with-opentelemetry branch from 5e8b123 to d11af79 Compare October 26, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.