Skip to content

fix(custom-metrics): roll forward to the new AI SDK - #223

Merged
sattensil merged 4 commits into
mainfrom
scarlett/sdk-roll-forward/custom-metrics
Oct 26, 2026
Merged

sattensil merged 4 commits into
mainfrom
scarlett/sdk-roll-forward/custom-metrics

Conversation

@sattensil

@sattensil sattensil commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Initialize with init_client / get_client; the complete workflow is async and ends with await shutdown().
  • New section on attributing custom metrics to a config variation with result.track_data.

Jira: DEVREL-1649

Testing

  • Not applicable
  • Manual (describe below)

Ran the complete workflow in custom-metrics.ipynb; the new metric appears in the metric list.

Notes

  • Targets launchdarkly-ai-server 0.3.0 (Python) and @launchdarkly/ai-node 0.3.0 (Node.js).
  • Part of a set of six roll-forward PRs (migrate, built-in-metrics, projects, tools, online-evals, custom-metrics). Each regenerates skills.json, so later merges may need python3 scripts/generate_catalog.py re-run.
  • The matching notebook in launchdarkly-labs/agentcontrol-cookbooks mirrors this skill's code; that update lands separately.

Note

Overview
Updates the custom-metrics agent skill (and matching skills.json entry to 1.1.0-experimental) so metric lifecycle guidance matches the new LaunchDarkly AI SDK and hosted MCP.

Create and read paths now lead with MCP (create-metric, get-metric, list-metrics, list-metric-events); update/delete stay on the REST API with a writer token. SDK examples switch from raw ldclient setup to launchdarkly-ai-server / @launchdarkly/ai-node (init_client, get_client, await shutdown() instead of flush()), with LD_SDK_KEY and prerequisites pointing at sdk-install.

Adds documentation for attributing custom metrics to config variations by passing track_data / trackData from config().invoke() into track(), plus an MCP-first create payload example. Compatibility text and doc links now reference the AI SDK references instead of the legacy Python SDK docs.

Reviewed by Cursor Bugbot for commit f418c1e. Bugbot is set up for automated code reviews on this repo. Configure here.

Initialize with init_client/get_client and LD_SDK_KEY, attribute custom metrics to config variations with the invoke() track data, and shut down with shutdown().
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 100/100 (4/4) no change
agentcontrol/configs-update 80/100 (4/5) 100/100 (5/5) +20
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
feature-flags/flag-and-release-change - 100/100 (4/4) new
feature-flags/flag-release - 80/100 (4/5) new
feature-flags/launchdarkly-flag-command - 100/100 (3/3) new
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (4/4) no change
feature-flags/launchdarkly-flag-drift - 100/100 (4/4) new
feature-flags/should-flag-change - 100/100 (17/17) new
onboarding - 100/100 (4/4) new

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

@sattensil
sattensil requested a review from jeffdupont October 23, 2026 20:17

@jeffdupont jeffdupont left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with one nit. Nothing PR-specific found; only the cross-cutting install note applies.


skills/agentcontrol/custom-metrics/SKILL.md:17 — Add launchdarkly-server-sdk to the Python prerequisites. The snippets import ldclient.Context, and launchdarkly-ai-server doesn't install it.

Use create-metric, get-metric, list-metrics, and list-metric-events; keep REST for update and delete, which have no MCP tool.
@sattensil
sattensil merged commit 8aead93 into main Oct 26, 2026
18 checks passed
@sattensil
sattensil deleted the scarlett/sdk-roll-forward/custom-metrics branch October 26, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants