Skip to content

Capture malformed tool-call entries and expose HAR extraction diagnostics #1681

Open

Description

Problem

HAR tool-call extraction tolerates malformed input by skipping it, but does not report which entries were rejected. Consumers cannot distinguish a complete extraction from a partial one.

PR #1488 demonstrates the gap: a non-object entry in a Chat Completions response's choices array is skipped so that valid calls can still be extracted. This prevents a crash, but produces no warning or rejected-entry record. Other extraction paths also skip malformed payloads.

We already preserve invalid JSON arguments on recognizable calls as arguments: { _raw: originalString }. That fallback does not capture malformed choices or call structures that cannot be represented as a ToolCall. The original HAR remains available, but investigating omissions requires manually inspecting it.

Desired behavior

Capture malformed tool-call-related entries as structured extraction diagnostics, without aborting extraction of valid calls or fabricating valid ToolCall objects.

  • Record the rejection reason and source location (HAR entry index, relevant choice/call/event index or JSON path, and transport/format where identifiable).
  • Preserve the rejected payload in a diagnostic artifact, or retain a reliable reference to it in the source HAR. Apply existing artifact access controls and sanitization; do not dump response bodies, arguments, or secrets into ordinary logs.
  • Expose and persist diagnostics alongside extraction results so downstream consumers can distinguish complete, partial, and failed extraction. Surface a concise warning/count when data is rejected.
  • Preserve the existing _raw argument fallback and successful tool-call extraction behavior.
  • Do not classify unrelated HTTP traffic, a legitimately empty response, or a valid response without tool calls as malformed.

Acceptance criteria

  • A response with choices: [null, validChoice] still yields the valid call and records a diagnostic for the null choice.
  • Malformed tool-call structures and malformed payloads in supported extraction paths produce actionable diagnostics rather than disappearing silently.
  • A valid call with invalid JSON arguments retains its raw arguments under the existing _raw behavior.
  • Valid responses without tool calls do not produce spurious malformed-call diagnostics.
  • Diagnostics remain accessible after extraction and identify the source data without leaking sensitive payloads into logs.
  • Focused regression tests cover mixed valid/malformed input and diagnostic persistence.

Related

Relevant code

  • packages/shared/src/har/har-parser.ts: extractToolCalls, processChoices, streaming and Anthropic extraction paths.
  • packages/shared/src/har/parsed-body.ts: body/SSE parsing currently discards malformed payloads.
  • packages/shared/src/har/responses-api-parser.ts: Responses API extraction.
  • packages/shared/src/har/types.ts: ToolCall currently has no associated extraction-diagnostics result.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions