Skip to content

fix(ai): capture usage and output of incomplete Responses streams - #983

Open
breken-ai wants to merge 1 commit into
PostHog:mainfrom
breken-ai:fix/responses-incomplete-stream-usage
Open

breken-ai wants to merge 1 commit into
PostHog:mainfrom
breken-ai:fix/responses-incomplete-stream-usage

Conversation

@breken-ai

Copy link
Copy Markdown

💡 Motivation and Context

A Responses API stream that is cut short, for example by max_output_tokens, ends on response.incomplete instead of response.completed. _ResponsesStreamState already reads the stop reason from any terminal event (#919), but extract_openai_usage_from_chunk and extract_openai_content_from_chunk only read response.completed.

As a result, these generations are captured with $ai_stop_reason: max_output_tokens but with no $ai_input_tokens/$ai_output_tokens and an empty $ai_output_choices, even though the tokens were billed and the partial output was streamed. Cost for truncated runs is missing from LLM analytics.

posthog-js's OpenAIResponsesStreamAccumulator takes usage and output from every terminal response. This PR does the same: usage and output are read from response.completed, response.incomplete and response.failed. A stream ends with exactly one of these, so nothing is counted twice.

💚 How did you test it?

  • New test test_streaming_responses_api_captures_usage_and_output_when_incomplete streams a real ResponseIncompleteEvent (status incomplete, incomplete_details.reason = max_output_tokens, usage 20/16, a partial output message) through the wrapped OpenAI client.
    • On main (4a138e6) it fails: KeyError: '$ai_input_tokens'.
    • With the fix it passes, including the partial output text.
  • pytest posthog 4144 passed, 16 skipped. ruff format --check, ruff check, mypy through mypy-baseline filter and python -W error -c "import posthog" are clean.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Fully autonomous

  • Found and fixed by an autonomous agent (Claude Code, Claude Opus) by comparing the Python OpenAI stream state with posthog-js's Responses accumulator. The failing test was run against main before the fix.
  • response.failed is included for parity with posthog-js and with the stop-reason handling. A failed response usually has no usage, and that case is already handled.

Commits here are unsigned; this repo squash-merges, so GitHub signs the merged commit.

A Responses API stream cut short, for example by max_output_tokens, ends on
response.incomplete instead of response.completed. The stream state only read
usage and output from response.completed, so these generations were captured
with no token counts and empty output even though the stop reason was set.

Read them from every terminal event (completed, incomplete, failed), as the
stop reason already does and as posthog-js does.
@breken-ai
breken-ai requested a review from a team as a code owner September 26, 2026 04:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant