Repository navigation
fix(ai): report OpenAI cache write tokens - #1268
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
SDK performance: posthog-pythonBenchmarks completed. No changes flagged for review. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate. View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
turnipdabeets
left a comment
There was a problem hiding this comment.
Thanks @alexeichhorn!
The cache_write_tokens tests added in #1268 were written before 8.0 moved AI events to capture_ai, so they read mock_client.capture and fail on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: a72b68c8-37a1-47ee-8b69-6c44f59250b2
test(ai): read cache write token assertions from capture_ai The cache_write_tokens tests added in #1268 were written before 8.0 moved AI events to capture_ai, so they read mock_client.capture and fail on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: a72b68c8-37a1-47ee-8b69-6c44f59250b2
💡 Motivation and Context
input_tokens_details.cache_write_tokens(Responses) orprompt_tokens_details.cache_write_tokens(Chat Completions). These tokens are part ofinput_tokensand are not incached_tokens. The OpenAI wrapper reads onlycached_tokens, so events never carry$ai_cache_creation_input_tokens, and ingestion prices cache writes as plain input tokens even thoughllm-costs.jsonhas acache_write_tokenrate for these models.cache_write_tokensalongsidecached_tokensin the converter's usage-field tables and map it tocache_creation_input_tokens. This covers Chat Completions and Responses, streaming and non-streaming, sync and async, because they all go through the same extractors. The shared capture code already emits$ai_cache_creation_input_tokenswhen the value is above zero. This matches@posthog/ai8.6.8 (fix(ai): report cache-write tokens for OpenAI-compatible providers posthog-js#4189).Out of scope
input_tokens_details.cache_write_tokens_details(ttl_30m_tokens/ttl_12h_tokens) is not mapped to$ai_cache_creation_5m_input_tokens/$ai_cache_creation_1h_input_tokens. Ingestion reads those properties only for Anthropic, and OpenAI's TTLs are 30m/12h rather than 5m/1h. The breakdown still reaches the backend in$ai_usage. Until ingestion prices 12-hour writes separately, they are costed at thecache_write_tokenrate.💚 How did you test it?
test_responses_cache_write_tokensandtest_chat_cache_write_tokens, each parametrized overstream=False/True. They use realopenaiSDK types through the wrapped client and assert$ai_input_tokens,$ai_cache_read_input_tokens, and$ai_cache_creation_input_tokenson the captured event. All four cases fail without the converter change.make_response_usagenow acceptscache_write_tokens.pytest posthog/test/aipasses, along withruff format --check,ruff check, mypy with the baseline filter, andpython -W error -c "import posthog". In the full suite, only three fork-based tests intest_metrics_autocapture.pyfail on macOS, and they fail the same way onmain.📝 Checklist
If releasing new changes
sampo add)🤖 Agent context
Autonomy: Human-driven (agent-assisted)
$ai_cache_creation_input_tokensproperty, and no new public API.input-costs.ts) before deciding against mapping the 30m/12h split. See "Out of scope".