[ccschmitz-launchdarkly]

EvaluationsModule.run now exports each dataset row as one trace, on the tracer provider init_client already registers. The $ld:ai:offline-evals:generation and :criterion events are unchanged.

Spec: launchdarkly/ai-sdks-monorepo#60 (TESTING.md §8.10). Frontend: launchdarkly/gonfalon#74117.

What each row exports

launchdarkly.dataset_row          root; gen_ai.conversation.id, launchdarkly.evaluation_run.id / .row_index
  chat {model}                    generation
  execute_tool {name}             one per recorded tool call
  evaluate {criterion}            gen_ai.evaluation.result, no verdict
    chat {model}                  judge call

• The root has no gen_ai operation, so the conversation view treats it as an envelope. Its status is ERROR when the handler raised.
• A provider handler traces its own call. For a plain function, the runner writes the chat and execute_tool spans itself. It does so only when the handler opened no SDK span, so nothing is traced twice.
• A row that didn't generate is never scored (§8.8.6), so it gets no evaluate span.
• Prompts, output, tool payloads, and judge explanations follow the existing capture_content gate, which is off by default. There is no separate evaluation switch.

Where the code lives

Core conversation.py gains two primitives that don't know about evaluations: stamp_span_attributes (write-if-absent attributes on SDK spans in a block) and count_sdk_spans. Everything evaluation-specific is in evaluations/spans.py.

Verified on staging

Project default, env ai-o-11-y, with a plain-function handler and the lookup_capital tool. Each run produced one trace with the five spans above and passed=1.

• Capture off: run · trace. The tree, model, usage, tool name, and score are present. No content attributes.
• Capture on: run · trace. Input and output messages, tool arguments and results, the judge prompt and completion, and the explanation are all present.

enable-evaluation-run-conversation-spans is still off in that environment, so the run page shows the synthetic waterfall. The trace links show the exported spans.

Known gaps

• The Python runner still emits a generation_incomplete criterion event for a failed row. §8.8.6 tracks that separately.
• gonfalon renders gen_ai.system_instructions as its raw JSON part array.

EvaluationsModule.run now exports each dataset row as one trace, on the tracer provider init_client already registers. The $ld:ai:offline-evals:generation and :criterion events are unchanged.

Spec: launchdarkly/ai-sdks-monorepo#60 (TESTING.md §8.10). Frontend: launchdarkly/gonfalon#74117.

What each row exports

launchdarkly.dataset_row          root; gen_ai.conversation.id, launchdarkly.evaluation_run.id / .row_index
  chat {model}                    generation
  execute_tool {name}             one per recorded tool call
  evaluate {criterion}            gen_ai.evaluation.result, no verdict
    chat {model}                  judge call

• The root has no gen_ai operation, so the conversation view treats it as an envelope. Its status is ERROR when the handler raised.
• A provider handler traces its own call. For a plain function, the runner writes the chat and execute_tool spans itself. It does so only when the handler opened no SDK span, so nothing is traced twice.
• A row that didn't generate is never scored (§8.8.6), so it gets no evaluate span.
• Prompts, output, tool payloads, and judge explanations follow the existing capture_content gate, which is off by default. There is no separate evaluation switch.

Where the code lives

Core conversation.py gains two primitives that don't know about evaluations: stamp_span_attributes (write-if-absent attributes on SDK spans in a block) and count_sdk_spans. Everything evaluation-specific is in evaluations/spans.py.

Verified on staging

Project default, env ai-o-11-y, with a plain-function handler and the lookup_capital tool. Each run produced one trace with the five spans above and passed=1.

Trace search defaults to the last 15 minutes, which no longer covers these runs. The trace links set startDate and endDate and do not set relativeTime. Each window is the root span, padded by 15 minutes on either side. The run page's conversation query omits its own window and lets the backend resolve the span range, so those links do not carry a time bound.

• Capture off: run · trace. Root span 2026-10-22T21:10:38.640Z, 1600467ns. The tree, model, usage, tool name, and score are present. No content attributes.
• Capture on: run · trace. Root span 2026-10-22T21:10:38.740Z, 2044814ns. Input and output messages, tool arguments and results, the judge prompt and completion, and the explanation are all present.

enable-evaluation-run-conversation-spans is still off in that environment, so the run page shows the synthetic waterfall. The trace links show the exported spans. Preview is gonfalon#74117 at c497293cd.

Known gaps

• The Python runner still emits a generation_incomplete criterion event for a failed row. §8.8.6 tracks that separately.
• gonfalon renders gen_ai.system_instructions as its raw JSON part array.

  

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4b9e771. Configure here.}

[jeffdupont]

Pushed f684f98 (merge of origin/main) and fa2e25e:

• Retitled to feat(evaluations): export each offline row as one trace.
• Both observer.count == 0 fallbacks now also need _sdk_processor_is_active() (new in lifecycle.py): true when _owns_otel_globals, or when the active provider carries ConversationIdSpanProcessor. Under a foreign provider the count stays 0 even when the handler traced itself, so the runner was writing every chat span twice. New test test_a_foreign_provider_gets_no_second_copy_of_handler_spans fails without the gate (4 chat spans instead of 2).
• module.py: ending the row spans is now in a try whose finally runs client.flush() and the tracer flush.

Trade-off: under a foreign provider, a handler that traces nothing gets no runner-written chat span. Needs a human reviewer.

[apucacao]

Approving based on the provided evidence. 👍 This is cool!