Skip to content

fix(http): handle leading UTF-8 BOM in SSE streams - #155

Open
monody0007 wants to merge 1 commit into
agentclientprotocol:mainfrom
monody0007:fix/sse-leading-utf8-bom
Open

monody0007 wants to merge 1 commit into
agentclientprotocol:mainfrom
monody0007:fix/sse-leading-utf8-bom

Conversation

@monody0007

Copy link
Copy Markdown

Summary

When an SSE response starts with a UTF-8 BOM (EF BB BF), parse_sse_stream decodes it as a leading U+FEFF. That makes the first field name \ufeffdata rather than data, so the first JSON-RPC event is silently dropped. This also happens when the BOM bytes arrive in separate chunks; later events still arrive.

Use the incremental utf-8-sig decoder to strip one leading BOM while preserving chunked decoding. The regression test covers complete and split BOMs, a Chinese payload, and a following event. Controls verify that a BOM inside JSON data remains intact and an invalid non-BOM byte still raises UnicodeDecodeError. This follows the WHATWG SSE parsing rule.

Related issues

Follow-up to #135, which added incremental UTF-8 decoding for split characters. The pre-submission search found no issue or open PR for the leading-BOM case.

Testing

  • Base revision 9d07d7871ef4b220b8507e15fc4b1560f0950a64, with the new tests copied into a separate checkout: the targeted command failed for both leading-BOM cases (2 failed, 3 passed, 8 deselected). Each failure showed only event 2 where events 1 and 2 were expected.
  • uv run --frozen python -m pytest tests/http/test_sse.py -k 'leading_bom or preserves_bom or rejects_invalid or multibyte_utf8' -q: 5 passed, 8 deselected.
  • uv run --frozen python -m pytest tests/http/test_sse.py tests/http/test_http_client.py -q: 22 passed.
  • make test: 361 passed, 1 skipped (optional Gemini CLI test).
  • make check: passed (pre-commit checks, Ruff, ty, and deptry).
  • git diff --check: passed.

Docs & screenshots

No public API, configuration, or schema changes. No documentation or screenshots needed for this parser correction.

Checklist

  • Conventional Commit title (fix:).
  • Tests cover the change.
  • Docs/examples considered; no update needed for this parser correction.
  • No schema regeneration required.

AI assistance disclosure: Codex helped prepare the code, tests, and description. AI tools ran the RED/GREEN tests and repository checks listed above, and an independent Opus model reviewed the change. No human code review is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant