Skip to content

OTel metrics have no service.instance.id, so concurrent sessions write to the same metric streams #5005

Description

@sebdanielsson

Describe the bug

Copilot CLI exports cumulative metrics (gen_ai.client.token.usage, github.copilot.tool.call.count, ...) with only service.name and service.version on the resource. There is no service.instance.id or any other attribute that is unique per process.

When several CLI processes export to the same backend, their metric streams have the same identity. That breaks the OTel single-writer rule. In Prometheus and Mimir, service.instance.id becomes the instance label, so every session writes to the same series. Each process starts its counters from zero, and the backend sees resets and out-of-order values. Sums and rates across users are wrong.

Affected version

GitHub Copilot CLI 1.0.88

Steps to reproduce the behavior

  1. COPILOT_OTEL_FILE_EXPORTER_PATH=/tmp/otel.jsonl copilot -p "Reply with: pong"
  2. Check the resource of the metric lines in /tmp/otel.jsonl. It only has service.name and service.version.
  3. Run two sessions at the same time with OTEL_EXPORTER_OTLP_ENDPOINT pointing at a collector that writes to Prometheus. Both sessions update the same series.

Expected behavior

Each process sets service.instance.id on the resource. A random UUID at startup is what the semantic conventions recommend. The session ID would also work.

Additional context

We can't work around this with resourceAttributes in enterprise managed settings, because that value is the same for every user. Right now we drop the CLI metrics in our collector and build our own metrics from the spans instead.

Linux (WSL2), x86_64.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions