Repository navigation
feat(mcp): tell directory reviews what the virtual tools record - #1204
Conversation
What: `get_more_tools` and `send_feedback` declare readOnlyHint, destructiveHint, idempotentHint and openWorldHint as false with no title (shared VIRTUAL_TOOL_HINTS). The injected `context` description is the short cross-SDK text. `llm_model` is never added to a schema's `required` list. Why: promised to OpenAI's plugin directory review. A call records an event in the server owner's project, so the old read-only/idempotent hints were wrong. Split from posthog-python#1199 so this has no public API change. Tested with: pytest posthog/test/mcp on the 1.x, v2 and fastmcp-2 venvs (670, 518, 665 passed); ruff format/check, mypy-baseline filter and `make public_api_check` clean (snapshot unchanged). Reviewer note: on the fastmcp-2 venv test_fastmcp_v2.py::test_jlowin_middleware_defaults_work_across_fresh_instances fails identically on origin/main, unrelated to this change.
There was a problem hiding this comment.
Approved.
The published tool contracts enter risky territory, covered by the trusted current-head review with no findings. The changes match the disclosed scope, and no substantive concerns or other refusal grounds were found.
- Author wrote 58% of the modified lines and has 5 merged PRs in these paths (familiarity MODERATE).
- PostHog Review (Standard) reviewed the current head: no findings.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 62L, 5F substantive, 170L/13F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1c-medium (170L, 13F, two-areas, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ f05a850 · reviewed head f05a850 |
posthog-python Compliance ReportDate: 2026-11-06T20:31:11.315427+00:00 ✅ All Tests Passed!121/121 tests passed Capture_V1 Tests✅ 95/95 tests passed View Details
Capture_Ai Tests✅ 5/5 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
Feature_Flags_Local_Evaluation Tests✅ 4/4 tests passed View Details
|
SDK performance: posthog-pythonBenchmarks completed. No changes flagged for review. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate. View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
What: removed the public PostHogMCP.capture_unknown_tool() and the public PostHogMCPAnalyticsEvent.UNKNOWN_TOOL constant; the event name now lives in the private event-name map. The registry helpers stay in _instrument_lowlevel next to _dispatch_can_differ instead of moving to _instrumentation. Dropped the slow-listing and hidden-tool test cases (they guarded a listing probe this branch no longer has and pass on main), the lowlevel test fixture edits, and README text that documented #1204's annotations. The manual-session test moved to test_posthog_mcp.py. Why: stamphog refused #1199 for a public API change and its size. The spec makes manual capture methods optional, so capture_unknown_tool can return in its own PR. Tested with: pytest posthog/test/mcp on .venv, v2, f2, v2f4 (689, 526, 681, 570 passed; f2's one failure is test_jlowin_middleware_defaults_work_across_fresh_instances, failing on main until #1203). ruff format/check, mypy-baseline filter clean. make public_api_check: snapshot up to date, identical to origin/main.
* feat(mcp): align posthog.mcp with the sdk-specs MCP spec before v1 What: - get_more_tools and send_feedback annotate readOnlyHint, destructiveHint, idempotentHint and openWorldHint as false, with no title, so the spec's exact annotations JSON matches. Hosts cannot override them. - The injected `context` argument uses the spec's shared description string. - `llm_model` is never added to a schema's `required` list. - A tools/call naming a tool the server never registered sends $mcp_unknown_tool (tool name, identity, session) instead of $mcp_tool_call and $exception. The client gets the same response. A disabled tool, invalid arguments, a raw server with no tools/list handler or a name missing from a paginated first page, and standalone FastMCP with dispatch-overriding middleware stay failed tool calls. PostHogMCP.capture_unknown_tool() covers custom dispatchers. - Every adapter's raise path and isError-result path calls one ToolCallLifecycle.record_if_unknown_tool(). The standalone FastMCP registry lookup and its middleware check moved to _instrumentation.py, so _instrument_v2 no longer imports from _instrument_lowlevel. - Minor sampo changeset. Why: posthog.mcp is about to be declared stable. Customers' insights over $mcp_* events must not shift after that, so every SDK matches the spec now. Tested: - pytest posthog/test/mcp: 678 passed, 10 skipped - annotations test compares the exact JSON; fails 2/4 with a title present - ruff format --check . and ruff check .: clean - mypy --no-site-packages --config-file mypy.ini . | mypy-baseline filter: clean - python -W error -c "import posthog": ok; make public_api_check: up to date - full pytest without provider API keys: 6142 passed, 3 failed in test_metrics_autocapture.py fork tests (macOS objc fork-safety abort, unrelated to this change) Reviewer notes: _standalone_tool_exists catches the cached fastmcp not-found errors, which include DisabledError; get_tool raises only NotFoundError in fastmcp 2.14 and 3.2, so a disabled tool still reads as existing. Client conversation ids are untouched (posthog-python#1193). * fix(mcp): close unknown-tool gaps found in review of the spec alignment What: - A disabled tool on a mounted FastMCP 3+ server stays a failed $mcp_tool_call. A server serving other providers (mounts, proxies) can't prove a name absent, so existence answers "can't tell". - FastMCP's stock middleware (logging, timing, error handling, retry, rate limiting, ping, auth, caching, response limiting) no longer hides the registry. Tool-injection middleware and app middleware still do. - A call naming no tool emits nothing, automatic or manual. - Truncated strings fit their limit with the "..." marker included, so $mcp_tool_name stops at 256 characters (the spec's Property reference). - first_listing_page reads nextCursor only when next_cursor is absent, so fastmcp 4 on MCP 2 logs no deprecation warning. - A raw server's unknown-tool error result carries the conversation handle again, and $mcp_unknown_tool carries it once delivered, as origin/main did and as posthog-js does. - Unknown-tool tests skip only the FastMCP.disable cases on fastmcp 2. - Changeset names the listing probe and hidden-tool upgrade differences. Why: posthog.mcp must match the sdk-specs MCP spec before it is declared stable, so customers' insights don't shift afterwards. Tested: - pytest posthog/test/mcp (mcp 1.30 + fastmcp 3.2): 694 passed, 11 skipped - test_unknown_tool.py on mcp 1.30 + fastmcp 2.14.7: 21 passed, 4 skipped; mcp 2.3: 15 passed, 10 skipped; mcp 2.3 + fastmcp 4.1.6: 25 passed - ruff format --check, ruff check, mypy | mypy-baseline filter, make public_api_check: clean Reviewer notes: posthog-js truncates with the marker past the limit too and needs the same fix. test_fastmcp_v2.py:: test_jlowin_middleware_defaults_work_across_fresh_instances fails on fastmcp 2.14.7 on origin/main as well ($mcp_input_keys is ["[redacted]"]); not touched here. * fix(mcp): bound the listing probe and give manual capture a session What: raw_listing_owns_tool_name waits at most LISTING_PROBE_DEADLINE_SECONDS (1s) for the host's tools/list and treats a timeout as None (can't tell), so a failed call records a failed $mcp_tool_call and the client gets its error promptly. PostHogMCP._base_event mints a fresh ses_<UUIDv7> when no session_id is passed, so capture_unknown_tool (and capture_tool_call and the other manual captures) no longer send no $session_id and an anonymous distinct_id. Why: PostHog Review on #1199. A stalled tools/list handler held every failed call forever; the spec's Session identity and Manual capture require a non-empty $session_id, fresh per call when nothing groups it. Tested: pytest posthog/test/mcp (697 passed, 11 skipped); ruff format/check, mypy | mypy-baseline filter, make public_api_check all clean. New tests failed first: stalled listing (timed out), manual capture without session (KeyError). Reviewer: a manual capture passing conversation_id but no session_id still gets a fresh session, not one derived from the handle. * fix(mcp): make the stalled-listing test stall on mcp 2.x What: `_raw(stall_listing=True)` now stalls every tools/list on mcp 2.x; on 1.x it still spares the server's own cache refresh. Why: the v2 adapter probes the host listing as `original(ctx, None)`, and the helper stalled only when a request was present, so the probe was answered at once, the call was recorded as `$mcp_unknown_tool`, and the test found no `$mcp_tool_call` (CI: MCP SDK v2 and v2-bare, Python 3.10). The product code was right: the deadline path falls back to a failed call on mcp 2. Tested: `pytest posthog/test/mcp` on mcp 1.30 + fastmcp 3.2 (697 passed), mcp 2.3 (536), mcp 2.3 + fastmcp 4.1.6 (579), and Python 3.10 with mcp 2.x bare (536) and with fastmcp 4 (579). Raising the deadline to 999s makes the test fail on mcp 2. ruff format/check and mypy baseline clean. * fix(mcp): stop treating a tools/list listing as proof a tool is unknown A call is now unknown only on authoritative evidence: jlowin FastMCP's registry when nothing overrides dispatch, or the official high-level server's registry. A raw low-level server's own dispatcher decides which names exist and its listing may hide tools it serves, so there the SDK answers "can't tell" and records a failed $mcp_tool_call (spec, "Tool call event"). This removes the failure-path listing probe, its one-second deadline, the pagination flag and FirstListingPage. The deadline also sat in the check that decides whether get_more_tools and send_feedback are virtual-tool calls, so a tools/list slower than one second sent those calls to the application. That check is back to origin/main's. Tested with: - new tests failed on the previous head: a slow tools/list still answers get_more_tools and send_feedback; a raw low-level server records a failed $mcp_tool_call for an unknown name and for a hidden tool - pytest posthog/test/mcp: .venv (mcp 1.30, fastmcp 3.2) 696 passed; v2 (mcp 2.3) 533 passed; v2f4 (mcp 2.3, fastmcp 4.1) 577 passed; f2 (mcp 1.30, fastmcp 2.14) 688 passed, 1 failed - ruff format --check, ruff check, mypy | mypy-baseline filter, make public_api_check: clean Reviewer notes: the f2 failure, test_jlowin_middleware_defaults_work_across_fresh_instances, also fails on origin/main in that venv. The unknown-tool handle-delivery test now covers the error-result branch through standalone FastMCP, since a raw low-level server no longer records unknown tools. * fix(mcp): drop the manual unknown-tool API and shrink the spec alignment What: removed the public PostHogMCP.capture_unknown_tool() and the public PostHogMCPAnalyticsEvent.UNKNOWN_TOOL constant; the event name now lives in the private event-name map. The registry helpers stay in _instrument_lowlevel next to _dispatch_can_differ instead of moving to _instrumentation. Dropped the slow-listing and hidden-tool test cases (they guarded a listing probe this branch no longer has and pass on main), the lowlevel test fixture edits, and README text that documented #1204's annotations. The manual-session test moved to test_posthog_mcp.py. Why: stamphog refused #1199 for a public API change and its size. The spec makes manual capture methods optional, so capture_unknown_tool can return in its own PR. Tested with: pytest posthog/test/mcp on .venv, v2, f2, v2f4 (689, 526, 681, 570 passed; f2's one failure is test_jlowin_middleware_defaults_work_across_fresh_instances, failing on main until #1203). ruff format/check, mypy-baseline filter clean. make public_api_check: snapshot up to date, identical to origin/main. * fix(mcp): treat FastMCP 2.x mounts as unable to rule out a tool Review: _serves_other_providers only checked FastMCP 3.x providers, so on 2.x a mounted server (or one whose mount failed to load) made a missing name look unregistered and emitted $mcp_unknown_tool instead of a failed tool call. It now also checks 2.x's _mounted_servers. Tested: new server-with-mounted-server case fails on fastmcp 2.14.7 before the fix and passes after; pytest posthog/test/mcp on fastmcp 3.2 (690 passed), 2.14.7 and mcp 2.3+fastmcp 4.1 (only failures are the stdio handshake tests, which fail locally without this change, and the 2.x input-keys test that posthog-python#1203 fixes); ruff clean.
Problem
ChatGPT's and Claude's plugin directory reviews flag servers instrumented with
posthog.mcp.get_more_toolsandsend_feedbackclaim to be read-only, idempotent and open-world, though each call records an event in the server owner's PostHog project, and the injectedcontextdescription asks for more text than review allows. The MCP analytics spec (sdk-specs, "Virtual tool annotations" and "Arguments the SDK injects") now requires them.Changes
get_more_toolsandsend_feedbackannotations:readOnlyHint,destructiveHint,idempotentHintandopenWorldHintare allfalse(VIRTUAL_TOOL_HINTS), with notitle.contextdescription is the shared text every PostHog MCP SDK sends: a few words of purpose, people by role, no quoting the user.llm_modelis never required. The unusedrequiredplumbing behind it is removed. A model the client sends in_metastill wins.This is the listing-only part of posthog-python#1199, split out because it adds no public API. The unknown-tool event stays in #1199.
How I tested
pytest posthog/test/mcp:test_jlowin_middleware_defaults_work_across_fresh_instances, fails on main too, and posthog-python#1203 fixes it.ruff format --check .,ruff check ., and mypy throughmypy-baseline filterare clean.make public_api_checkpasses with the snapshot unchanged.