Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ The client manages a singleton connection to LaunchDarkly and the associated tel
|---|---|
| `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. Does not take behavior defaults. |
| `init_client(client=...)` | **BYOC overload** — accepts a pre-initialized `LDClientInterface`. Stores it directly without calling the SDK. Flushes pending `$ld:ai:sdk:info` events. |
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = False`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = True`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it, and an explicit `False` strips content everywhere, including offline evaluation spans. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
| `get_client()` | Returns the initialized `LDClientInterface`. Throws if initialization has not completed. |
| `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. Restores the built-in SDK defaults. |
| `register_ai_sdk_package(name, version)` | Records a LaunchDarkly AI package identity. Handler and convenience packages call this at import time. |
Expand Down Expand Up @@ -571,14 +571,15 @@ beside the input count, so its handlers add it in. OpenAI and LangChain already
input, so theirs pass the figure through. Centralising that rule would double-count for two
providers out of three. `SpanUsage` is the type that means the folding is already done.

#### Content is off by default
#### Content is on by default

Every factory takes `capture_content: bool | None = None`. `None` reads `configure()` at
invocation, which is off until a process sets `SDKOptions(capture_content=True)`. An explicit
`True` or `False` overrides that default. Guard at the call site as well as inside the helper: the
helper's guard makes a forgotten call site harmless, and the call site's guard avoids serialising
JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the handler
runs, not when it is constructed.
invocation. The built-in default is on until a process sets `SDKOptions(capture_content=False)`.
An explicit `True` or `False` overrides that default, and an explicit `False` strips content
everywhere, including offline evaluation spans. Guard at the call site as well as inside the
helper: the helper's guard makes a forgotten call site harmless, and the call site's guard avoids
serialising JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the
handler runs, not when it is constructed.

#### Finish reasons have three mechanisms, not one

Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -542,20 +542,23 @@ This matters most on Anthropic, which reports cache reads and writes beside the
than inside it. A turn that reads 19,971 tokens from cache and writes 3,580 more reports an input of
3 from the provider; the span reports 23,554.

### Conversation content is off by default
### Conversation content is on by default

Prompts, model output, tool arguments and tool results are personal data, so a span carries only
metadata unless you ask for more: models, token counts, timings, tool names.
Prompts, model output, tool arguments and tool results are personal data. The built-in default
sends that text. A span carries only metadata — models, token counts, timings, tool names — when
capture is turned off.

Pass `capture_content=True` to a handler factory to include the conversation, or turn it on for every handler from process startup. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).
Pass `capture_content=False` to a handler factory to leave the conversation off that handler, or
turn it off for every handler from process startup. An explicit `False` wins everywhere, including
on offline evaluation spans. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).

```python
from launchdarkly_ai_claude_messages import create_claude_messages_handler

handler = create_claude_messages_handler(capture_content=True)
handler = create_claude_messages_handler(capture_content=False)
```

Turning this on sends the text of every request and response to whatever collector the SDK points
Leaving this on sends the text of every request and response to whatever collector the SDK points
at.

### Streaming
Expand Down
25 changes: 14 additions & 11 deletions TELEMETRY-CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ able to tell from the trace which path ran.
| `launchdarkly.stream.abandoned` | `True`, only when abandoned | `end_span_once` |
| `gen_ai.evaluation.name` | judge config key, judge roots only | `with_judge_evaluation`, see section 4a |
| `gen_ai.evaluation.score.value` | numeric score, judge roots only | `with_judge_evaluation`, see section 4a |
| `gen_ai.evaluation.explanation` | judge reasoning, judge roots only, gated by section 7 | `with_judge_evaluation`, see section 4a |

The root also carries one span event, `feature_flag`, with these event attributes:

Expand Down Expand Up @@ -226,17 +227,18 @@ SDK writes a `gen_ai.evaluation.result` span event on that `invoke_agent` span:
|---|---|
| `gen_ai.evaluation.name` | judge config key |
| `gen_ai.evaluation.score.value` | numeric score, only when the judge returned a finite number |
| `gen_ai.evaluation.explanation` | the judge's reasoning, gated. See section 7 |

The same keys are mirrored as span attributes, so section 2 lists them too.
`gen_ai.evaluation.score.label` is not invented.
The existing `track(evaluationMetricKey)` call is unchanged and still feeds AI Config Monitoring —
a judge that returns a non-numeric score emits no evaluation event but still tracks the metric.

`gen_ai.evaluation.explanation` is deliberately **not** emitted. The judge's reasoning is
model-generated prose about the user's conversation — content, under section 7 — and content
attributes require `captureContent` / `capture_content`, a handler-factory option this layer does
not receive. The reasoning is still returned to the caller in `judgeResults` / `judge_results`;
only the telemetry copy is withheld. Exporting it needs its own opt-in.
`gen_ai.evaluation.explanation` is content under section 7. The judge's reasoning is
model-generated prose about the user's conversation, so it follows the judge handler's own
`capture_content` decision. It is written by default, and an explicit `False` on the judge's
handler or the process default withholds it. The reasoning is always returned to the caller in
`judgeResults` / `judge_results`; only the telemetry copy is gated.

---

Expand Down Expand Up @@ -397,12 +399,13 @@ error for an abandoned stream, so two dashboards would describe the same run dif
Every handler factory takes `capture_content: bool | None = None`.

`None` reads the process-wide default at invocation (`configure(options=SDKOptions(...))`), which
is off. `True` or `False` overrides that default for the handler, and an explicit `False` wins
over a configured `True`. A convenience wrapper's `capture_content` is the per-call override: when
the caller passes it, that value is what the handler is built with; when they omit it, the wrapper
passes `None` and the process default applies. Conversation content is personal data. A run emits
only metadata, meaning models, token counts, timings and tool names, until capture is turned on.
Turning it on sends the text of every request and response to whatever collector the SDK points at.
is on. `True` or `False` overrides that default for the handler, and an explicit `False` wins
over a configured `True`, including on offline evaluation spans. A convenience wrapper's
`capture_content` is the per-call override: when the caller passes it, that value is what the
handler is built with; when they omit it, the wrapper passes `None` and the process default
applies. Conversation content is personal data. The default sends the text of every request and
response to whatever collector the SDK points at. An explicit `False` emits only metadata, meaning
models, token counts, timings and tool names.

Guard twice, on purpose. The helper returns without writing when capture is off, which makes a
forgotten call site harmless. The call site also checks before building the argument, which avoids
Expand Down
16 changes: 9 additions & 7 deletions examples/capture_content.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
"""
Example: enable conversation content for every handler at process startup.

The handlers in the global registry are built with no ``capture_content``
argument. They read the process default when they run. ``configure`` is
separate from ``init_client``: ``main.py`` initializes the client, and this
example calls ``configure`` once before the first AI invocation. An explicit
``True`` or ``False`` on a factory would override this default.
Example: pin conversation content on for every handler at process startup.

The built-in default is already on. The handlers in the global registry are
built with no ``capture_content`` argument and read that default when they
run. ``configure`` is separate from ``init_client``: ``main.py`` initializes
the client, and this example calls ``configure`` once before the first AI
invocation so the choice is visible at startup. Pass
``SDKOptions(capture_content=False)`` to opt out. An explicit ``False`` on a
factory removes the transcript everywhere, including offline evaluation spans.

Usage (via main.py):
python main.py capture-content <flag-key> "<user input>"
Expand Down
8 changes: 4 additions & 4 deletions examples/conversation.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
the same id: root, ``chat``, ``execute_tool``, and the judge's own ``invoke_agent``.
2. Each turn shows a score badge, sourced from the ``gen_ai.evaluation.result`` span event on
the judge span.
3. No judge reasoning anywhere in the telemetry. The score and the judge's config key are
exported; the explanation is not, because it is model prose about the user's conversation
and content attributes require ``capture_content``. The reasoning IS printed below, straight
from ``judge_results`` — that is the caller's copy, and it is unaffected.
3. The judge explanation is on the span, because capture is on by default. The score and
the judge's config key are exported either way. An explicit ``False`` removes the
explanation from telemetry. The reasoning IS printed below, straight from
``judge_results`` — that is the caller's copy, and it is unaffected.

The flag key must point at an AI Config with a ``judge_configuration``, otherwise there are no
judge turns to look at.
Expand Down
2 changes: 1 addition & 1 deletion packages/claude-agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ asyncio.run(main())

| Option | Default | Description |
|---|---|---|
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is off by default. |
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is on by default. An explicit `False` leaves them off. |
| `cwd` | the process's working directory | Working directory for the Claude Code process the Agent SDK starts. A `str` or path-like. |
| `setting_sources` | `[]` (none) | Which Claude Code settings sources to load: `"user"` (`~/.claude/settings.json`), `"project"` (`.claude/settings.json`, and `CLAUDE.md`), `"local"` (`.claude/settings.local.json`). |

Expand Down
2 changes: 1 addition & 1 deletion packages/claude-agents/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -739,8 +739,8 @@ def create_claude_agents_handler(

Set *capture_content* to put prompts, model output, tool arguments and tool results on the
emitted spans. ``None`` (the default) reads the process-wide default at invocation,
which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default
for this handler. See TELEMETRY-CONTRACT.md section 7.
which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default
for this handler. An explicit ``False`` emits only metadata. See TELEMETRY-CONTRACT.md section 7.

*cwd* and *setting_sources* configure the Claude Code process the Agent SDK starts, not the
model, so they come from the application and never from a config. Each one given is passed to
Expand Down
34 changes: 28 additions & 6 deletions packages/claude-agents/tests/test_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -862,7 +862,25 @@ async def test_cache_tokens_folded_into_input(


class TestContentCapture:
async def test_emits_no_content_at_all_by_default(
async def test_emits_content_by_default(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
handler_mod,
"query",
_fake_query(
[
assistant_message(content=[TextBlock(text="hi")]),
result_message("hi"),
]
),
)
await create_claude_agents_handler()(BASE_CONFIG, "q")
[chat] = named("chat ")
assert "gen_ai.input.messages" in chat.attributes
assert "gen_ai.output.messages" in chat.attributes

async def test_emits_no_content_when_capture_content_is_false(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
Expand All @@ -872,22 +890,22 @@ async def test_emits_no_content_at_all_by_default(
[init_message(tools=["Read"]), assistant_message(), result_message()]
),
)
await create_claude_agents_handler()(TOOL_CONFIG, "q")
await create_claude_agents_handler(capture_content=False)(TOOL_CONFIG, "q")
for span in (root(), *named("chat ")):
assert "gen_ai.input.messages" not in span.attributes
assert "gen_ai.output.messages" not in span.attributes
assert "gen_ai.system_instructions" not in span.attributes
assert "gen_ai.tool.definitions" not in span.attributes

async def test_root_carries_no_content_by_default_either(
async def test_root_carries_no_output_when_capture_is_false(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
handler_mod,
"query",
_fake_query([assistant_message(), result_message("hi")]),
)
await create_claude_agents_handler()(BASE_CONFIG, "q")
await create_claude_agents_handler(capture_content=False)(BASE_CONFIG, "q")
assert "gen_ai.output.messages" not in root().attributes

async def test_tool_call_arguments_and_result_gated_by_capture(
Expand Down Expand Up @@ -925,7 +943,7 @@ async def _query(**kwargs: Any) -> AsyncIterator[Any]:
yield result_message("done")

monkeypatch.setattr(handler_mod, "query", _query)
await create_claude_agents_handler()(
await create_claude_agents_handler(capture_content=False)(
TOOL_CONFIG, "q", {"search": lambda _: "r"}
)
[tool_span] = named("execute_tool ")
Expand Down Expand Up @@ -1610,7 +1628,11 @@ async def test_ld_span_attributes_land_on_root_only(
assert attrs["launchdarkly.config.key"] == "cfg"
assert attrs["launchdarkly.variation.key"] == "var"
assert attrs["launchdarkly.run.id"] == "run-1"
assert [e.name for e in root().events] == ["feature_flag"]
assert [
event.name
for event in root().events
if not event.name.startswith("gen_ai.content")
] == ["feature_flag"]


class TestModelParametersForwarding:
Expand Down
2 changes: 1 addition & 1 deletion packages/claude-messages/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -537,9 +537,9 @@ def create_claude_messages_handler(

Set *capture_content* to put prompts, model output, tool arguments and tool results on the
emitted spans. ``None`` (the default) reads the process-wide default at invocation,
which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default
for this handler. Conversation content is PII, so a run emits only metadata, meaning
models, token counts, timings and tool names, until capture is turned on. Turning this
which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default
for this handler. Conversation content is PII. An explicit ``False`` emits only metadata,
meaning models, token counts, timings and tool names. Turning this
on sends the text of every request and response to whatever collector the SDK is pointed at.
"""
report_usage(
Expand Down
Loading
Loading