fix(semconv): dual-emit OTel GenAI input/output token usage names

[ZxlDragonDoctor]

Problem

AgentOps still emits the older GenAI semconv usage names (gen_ai.usage.prompt_tokens / completion_tokens) only. Current OTel GenAI conventions use gen_ai.usage.input_tokens / output_tokens. In a collector that mixes AgentOps with OpenLLMetry / pydantic-ai spans, sum(input_tokens) misses AgentOps and sum(prompt_tokens) misses the others.

Fixes the transition gap described in #1447. Addresses the migration question: this PR dual-emits both families instead of a breaking rename, so existing AgentOps dashboards keep working while generic OTel consumers can sum the current names.

Solution

• Add SpanAttributes.LLM_USAGE_INPUT_TOKENS / LLM_USAGE_OUTPUT_TOKENS.
• In TokenUsage.to_attributes(), emit the current OTel names alongside the legacy names when prompt/completion counts are present.
• Keep zero-suppression behavior unchanged (no empty usage fields on non-LLM spans).
• Extend unit coverage for the dual-emit path.

No call-site rewrite in this PR — every existing setter of the legacy constants benefits automatically.

Testing

• Offline assertions mirroring TokenUsage.to_attributes() dual-emit logic: token_usage_dual_emit: 3 passed
• Extended tests/unit/instrumentation/common/test_token_counting.py:
  • zero values still produce {}
  • positive prompt emits both prompt_tokens and input_tokens
  • completion emits both completion_tokens and output_tokens

Agent dimension

evaluation

Core value

Let mixed-SDK OTel collectors sum AgentOps token usage with the current GenAI attribute names without breaking existing prompt/completion consumers.

Fixes #1447

[roy-tong]

Reviewed — this implements the plan from #1447 cleanly, and the test asserting both families (and the negative case) is exactly right. Approving with three non-blocking notes:

1. Double-count hazard should be stated where a future reader will meet it. The code comment says collectors "can sum either family" — the sharper warning is that a consumer summing both families counts twice. One sentence in the span_attributes.py comment block ("consumers must prefer the new names; legacy names are compatibility aliases — summing both double-counts") would prevent the predictable dashboard bug.

2. Internal consumers worth a quick check. If anything on the ingestion side iterates gen_ai.usage.* attributes and sums matches, dual-emit doubles it internally the same way. The unit tests cover to_attributes()'s output shape, which suggests no such iteration exists — just worth confirming once.

3. Removal horizon, even soft. "During the transition (#1447)" is good; if there's a version where the legacy names are planned to drop, naming it here (or in a tracking issue linked from this comment) gives dashboard owners a date to migrate by. The failure mode is silent: dashboards keyed on prompt_tokens break quietly the day the alias is removed.

One forward-looking note, not for this PR: cache_creation_input_tokens / cache_read_input_tokens above are the same generation of legacy names on the cache side — the current conventions nest them (gen_ai.usage.cache_read.input_tokens etc.), so the same dual-emit pattern will be needed there eventually, and it's the exact alias-drift shape that bit other SDKs' extractors (langfuse#17117 for one). The regression shape that guards the whole family: same logical call, two attribute names, one expected total.

[ZxlDragonDoctor]

Friendly ping - this PR was approved on 2026-09-15 and is still open. Happy to rebase on main or adjust anything needed for merge. Thanks!

[ZxlDragonDoctor]

Friendly ping - this PR was approved on 2026-09-15 and is still open. Happy to rebase on main or address any remaining non-blocking notes if that helps merge. Thanks!

[ZxlDragonDoctor]

Friendly ping — this PR is still open and ready for review. Happy to rebase if that helps merge. Thanks!

[ZxlDragonDoctor]

Friendly ping — this PR is still open and ready for review. Happy to rebase if that helps merge. Thanks!

[roy-tong]

Please go ahead with the rebase — much appreciated, and sorry for leaving it stale. The branch head is roy-tong/AgentMeasure@2cf476d if that helps the rebase; nothing else has changed on our side since your last review. Once it's rebased we'll watch CI and turn any red around same-day.

[ZxlDragonDoctor]

Rebased as requested - thanks for the nudge.

• Rebased onto current main (f8e907b); the branch was already up to date with main (0 commits behind), so the content is unchanged - I refreshed the commits and force-pushed so the workflows re-trigger (db5a874).
• Local verification on the rebased head: pytest tests/unit/instrumentation/common/ -q ? 57 passed (includes test_dual_emits_current_otel_genai_names).

One note on CI: the three workflows (Python Tests, Static analysis, Examples Integration Test) are currently in action_required state - GitHub is waiting for a maintainer to click Approve and run on this fork PR. Once approved, they should go green on the new head.

[roy-tong]

Thank you — saw the rebase (head db5a874, mergeable, no conflicts). We'll watch CI and turn any red around same-day. Appreciate you carrying it over the line.