Repository navigation
Conversation
Rewrite the migration around config().invoke() with handler packages or custom handlers, keeping the original call as the fallback. Remove tracker wiring (the SDK records metrics), switch placeholders to {{name}} without spaces, pass tools as tool_handlers, and update the agent-mode, agent-graph, fallback, audit, and before/after references for launchdarkly-ai-server 0.3.0 and @launchdarkly/ai-node 0.3.0.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 471a3a7. Configure here.
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
Raise a dedicated exception at the ReAct step limit so the RuntimeError fallback doesn't swallow it, accept sync or async tool functions in the StateGraph example, and drop tools from parameters in the LangChain factory example.
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Its core fallback, except RuntimeError, also catches failures from the SDK's own handlers after they have already run tools. Its install step leaves out launchdarkly-server-sdk, so every request silently takes the fallback.
skills/agentcontrol/migrate/references/phase-1-analysis-checklist.md:143 — launchdarkly-ai-server doesn't bundle the server SDK. On PyPI its only required dependency is opentelemetry-api, and none of the handler packages pull it in either. Without launchdarkly-server-sdk, init_client() raises RuntimeError("LaunchDarkly server SDK not installed...") (python-ai-sdk lifecycle.py:271-280). In this skill that error lands in the except RuntimeError fallback, so a migrated app silently runs the hardcoded path on every request and the Monitoring tab stays empty. Suggest: launchdarkly-ai-server + launchdarkly-server-sdk (required peer). Same fix for the install list at SKILL.md:160. The ai-node row is correct as written.
skills/agentcontrol/migrate/SKILL.md:406 — "Catch RuntimeError specifically; provider errors are different exception types" doesn't hold for the handler packages this skill recommends. They raise RuntimeError themselves after the run has started:
Tool loop exceeded the maximum number of steps (10)in openai-messages, claude-messages and langchain-messagesClaude agent run ended with ...in claude-agents
The SDK re-raises these unchanged. So an agent that hits the step cap after calling a side-effecting tool falls into this fallback, which re-runs the hardcoded agent and calls the tool again. The AgentStepLimitError fix in 71ce24e covers only the custom-loop example.
Smallest fix, on the error path only: inspect_config never raises, so re-raise when the config was actually served:
except RuntimeError:
if (await inspect_config("chat-assistant", context))["enabled"]:
raise
return answer_hardcoded(user_input)The same pattern applies at SKILL.md:24, SKILL.md:234, references/fallback-defaults-pattern.md:9,42, references/before-after-examples.md:75,221 and references/agent-graph-reference.md (the graph except (RuntimeError, ValueError)). For Node.js (fallback-defaults-pattern.md:60, before-after-examples.md:155), "anything that isn't a provider error" also catches Error('Tool loop exceeded...') and tool-handler errors. The same inspectConfig check works there.
skills/agentcontrol/migrate/SKILL.md:267 — On launchdarkly-ai-server 0.4.0, a list works too: handler_key() normalizes list and tuple tags (python-ai-sdk #152). Only 0.3.0 needed a tuple. Every handler package on PyPI requires launchdarkly-ai-server>=0.3.1, so anyone who installs a handler today gets 0.4.0. Suggest dropping this sentence, keeping the tuple in the example, and bumping the compatibility line to 0.4.0+.
skills/agentcontrol/migrate/SKILL.md:421 — This contradicts the caveat at :241. On 0.2.5, openai-messages forwards no parameters and claude-messages forwards only max_tokens, so neither can crash on max_search_results. Only the LangChain handler or a custom handler passes raw params through. python-ai-sdk #107 (merged, unreleased) and js-ai-sdk #118 change every handler to forward a fixed allowlist and drop other keys. The advice to use model.custom stays right, but the reason should be "handlers won't pass it to your code, and custom/LangChain handlers would send it to the provider." Suggest tying the :241 caveat to versions: "openai-messages ≤ 0.2.5 / claude-messages ≤ 0.2.5; fixed by python-ai-sdk #107 in the next release." The same rewording applies at agent-mode-frameworks.md:86, phase-1-analysis-checklist.md:108 and before-after-examples.md:88.
| messages = [ | ||
| {**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", []) | ||
| ] | ||
| response = openai_client.chat.completions.create( |
There was a problem hiding this comment.
Should fix. call_openai is async, but it calls the sync openai_client.chat.completions.create(...). Every request holds the event loop for the whole model call, so concurrent requests in an async server run one at a time. before-after-examples.md:59 already uses AsyncOpenAI with await. Use the same here: response = await async_openai.chat.completions.create(...).
There was a problem hiding this comment.
Fixed in 31e10a3. The example creates an AsyncOpenAI client and awaits the call.
| 3. **Build a `Context` per request.** The context carries targeting inputs — user ID, plan tier, region, whatever the rollout is keyed on. Reuse the same context the app already uses for feature flag evaluation if one exists. | ||
| 4. **Always check `config.enabled`.** Even a successful `completion_config` call can return a disabled config (if the variation is turned off in LaunchDarkly). The disabled path should not call the provider. | ||
| 5. **Do not cache the config object across requests.** Call `completion_config` / `agent_config` inside the request handler so LaunchDarkly can re-evaluate targeting per call. One `LDAIClient` instance, many `completion_config` calls. | ||
| 1. **The request handler's signature and return shape stay the same.** Only where the model, prompt, and parameters come from changes, plus the `try` / `except RuntimeError` around `invoke()`. |
There was a problem hiding this comment.
Should fix. Rule 1 says the request handler's signature stays the same, and SKILL.md:92 says the same. But every example turns a sync answer(user_question) into async def answer(user_id, user_question) (:18 → :71, and SKILL.md:210 → :230). Every existing caller breaks: a sync caller now gets a coroutine back, and every caller has to supply user_id. An agent following the rule won't update the callers. Suggest saying it plainly: "The call site becomes async and needs a context source (user id); update every caller, or keep a sync wrapper with asyncio.run for scripts."
There was a problem hiding this comment.
Fixed in 31e10a3. Core principle 2 and rule 1 now say the call site becomes async and needs a context source, and to update every caller or keep an asyncio.run() wrapper for scripts.
| tracker.track_judge_result(result) | ||
| result = await config(key="chat-assistant", handler=handlers, skip_judges=True).invoke(user_input, context) | ||
| for task in result.judge_tasks or []: | ||
| asyncio.create_task(run_judge_worker(task)) |
There was a problem hiding this comment.
Should fix. The event loop keeps only a weak reference to tasks, so a fire-and-forget create_task(run_judge_worker(task)) can be garbage-collected before it finishes, and that judge score is lost. The asyncio docs carry this warning for create_task. Smallest fix: keep the tasks in a set, as in the docs: t = asyncio.create_task(...); _bg.add(t); t.add_done_callback(_bg.discard). Or point at the online-evals worker instead of inlining this.
There was a problem hiding this comment.
Fixed in 31e10a3. The tasks go into a set with add_done_callback(discard), as in the asyncio docs.
| 4. **Wire feedback tracking if the app has thumbs-up/down UI.** Both SDKs expose `trackFeedback` with a `{kind}` argument. | ||
| 1. **Check the Monitoring tab.** Hit the wrapped endpoint in staging, then open the config in LaunchDarkly → **Monitoring**. Duration, token, and generation counts should appear within 1–2 minutes. Force an error (bad provider key) and confirm the error count increments; the SDK records the error and re-raises. | ||
| 2. **If tokens are missing**, the handler isn't returning usage. Custom handlers must return `{"output": ..., "usage": {"input": n, "output": n}}`. For streaming, enable usage on the final chunk (`stream_options={"include_usage": True}` for OpenAI). | ||
| 3. **Wire feedback if the app has thumbs-up/down UI.** The SDK doesn't record feedback. Store the result's `track_data` with the message and send the event when the user responds: |
There was a problem hiding this comment.
Nice to have. Neither SDK's stream() done event carries track_data today: Python yields response, usage and judge_results (client.py:272-277), and JS yields response, usage and judgeResults. A streaming app with thumbs-up/down has nothing to attribute feedback with. Worth one sentence saying so, since the coverage table sends streaming apps to stream().
There was a problem hiding this comment.
Added in 31e10a3. The feedback step now says stream() doesn't return track_data yet.
…judges Use AsyncOpenAI in the custom handler, say plainly that call sites become async and need a context source, keep references to background judge tasks, note that stream() has no track_data, and set custom knobs with the MCP variation tool instead of curl.
Summary
config().invoke()with handler packages or a custom handler, keeping the original provider call as the fallback wheninvoke()raisesRuntimeError.{{name}}with no spaces;ldctx→ldContext. Tools move totool_handlers.graph(),resolve_graph,to_lang_graph).Jira: DEVREL-1644
Testing
Ran before/after Example 1 against a live config: the served path and the no-SDK-key fallback path both returned. Checked every API name, signature, and package against the installed SDKs; all links return 200.
Notes
launchdarkly-ai-server0.3.0 (Python) and@launchdarkly/ai-node0.3.0 (Node.js).skills.json, so later merges may needpython3 scripts/generate_catalog.pyre-run.Note
Overview
Rolls the AgentControl
migrateskill forward to the current AI SDKs (launchdarkly-ai-server/@launchdarkly/ai-node0.3.0+), bumping the skill to 0.2.0 inskills.jsonand rewriting the five-stage flow and all migrate reference docs.The workflow now centers on
config().invoke()with handler packages or custom handlers,tool_handlersfor tools, and{{name}}placeholders (no spaces). Stage 4 is verify metrics instead of wiringcreate_tracker()/track_*— the SDK records duration, tokens, success/error, and tool calls automatically. Fallbacks are app-owned: keep the original hardcoded call and run it onRuntimeError, replacing*ConfigDefaultpassed intocompletion_config/agent_config.Adds explicit guidance for migrating off the legacy AI SDK (
ldai,completion_config, trackers), movesconfigs-targetingimmediately after config creation, and refreshes LangGraph/CrewAI/Strands patterns, before/after examples, fallback docs, metrics mapping, and agent graphs (graph(),resolve_graph,to_lang_graph) without manual graph trackers.Reviewed by Cursor Bugbot for commit 31e10a3. Bugbot is set up for automated code reviews on this repo. Configure here.