Repository navigation
fix(custom-metrics): roll forward to the new AI SDK - #223
Merged
Merged
Conversation
Initialize with init_client/get_client and LD_SDK_KEY, attribute custom metrics to config variations with the invoke() track data, and shut down with shutdown().
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
jeffdupont
approved these changes
Oct 26, 2026
jeffdupont
left a comment
There was a problem hiding this comment.
Approving with one nit. Nothing PR-specific found; only the cross-cutting install note applies.
skills/agentcontrol/custom-metrics/SKILL.md:17 — Add launchdarkly-server-sdk to the Python prerequisites. The snippets import ldclient.Context, and launchdarkly-ai-server doesn't install it.
Use create-metric, get-metric, list-metrics, and list-metric-events; keep REST for update and delete, which have no MCP tool.
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
init_client/get_client; the complete workflow is async and ends withawait shutdown().result.track_data.Jira: DEVREL-1649
Testing
Ran the complete workflow in
custom-metrics.ipynb; the new metric appears in the metric list.Notes
launchdarkly-ai-server0.3.0 (Python) and@launchdarkly/ai-node0.3.0 (Node.js).skills.json, so later merges may needpython3 scripts/generate_catalog.pyre-run.Note
Overview
Updates the custom-metrics agent skill (and matching skills.json entry to
1.1.0-experimental) so metric lifecycle guidance matches the new LaunchDarkly AI SDK and hosted MCP.Create and read paths now lead with MCP (
create-metric,get-metric,list-metrics,list-metric-events); update/delete stay on the REST API with a writer token. SDK examples switch from rawldclientsetup tolaunchdarkly-ai-server/@launchdarkly/ai-node(init_client,get_client,await shutdown()instead offlush()), withLD_SDK_KEYand prerequisites pointing atsdk-install.Adds documentation for attributing custom metrics to config variations by passing
track_data/trackDatafromconfig().invoke()intotrack(), plus an MCP-first create payload example. Compatibility text and doc links now reference the AI SDK references instead of the legacy Python SDK docs.Reviewed by Cursor Bugbot for commit f418c1e. Bugbot is set up for automated code reviews on this repo. Configure here.