Skip to content

fix(workers-ai-provider): read each streamed piece once when Llama sends it twice - #691

Open
Plopix wants to merge 1 commit into
cloudflare:mainfrom
Plopix:fix/llama-stream-read-once
Open

Plopix wants to merge 1 commit into
cloudflare:mainfrom
Plopix:fix/llama-stream-read-once

Conversation

@Plopix

@Plopix Plopix commented Nov 18, 2026 •

Copy link
Copy Markdown

Summary

Llama models on Workers AI stream a hybrid event: the same piece is in the native fields and in the OpenAI-style delta of the same event.

data: {"response":" Canberra.","tool_calls":[],"p":"…","choices":[{"index":0,"delta":{"role":"assistant","content":" Canberra."}}]}
data: {"response":"","tool_calls":[{"arguments":"lk\"}"}],"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":null,"function":{"arguments":"lk\"}"}}]}}]}

getMappedStream reads both, so text comes out twice and tool-call arguments are concatenated twice:

 on(event)
   text
-    emit response;  emit choices[0].delta.content
+    emit response, or choices[0].delta.content when response is empty
   tool calls
-    emit tool_calls;  emit choices[0].delta.tool_calls
+    emit choices[0].delta.tool_calls (it has index and id), or tool_calls when the delta has none

Streams that use only one of the two shapes take the same path as before.

createAI is not affected. createAI from agents/models/ai-sdk (0.33.0), the adapter the README now recommends for the binding, reads text and tool calls only from choices[0].delta, so it reads each piece once. Checked against the live binding: with llama-3.3-70b-instruct-fp8-fast and llama-4-scout-17b-16e-instruct, createAI + streamText returned "The capital of Australia is Canberra." once and a valid add_to_list call ({"items": ["milk"]}). This PR fixes the paths that still go through getMappedStream: credentials mode (accountId + apiKey, not deprecated), the AI Search chat model, and existing createWorkersAI({ binding }) users.

Evidence

  • Before: streamText through the binding with @cf/meta/llama-4-scout-17b-16e-instruct:

    • the answer reads "TheThe capital capital of of Australia Australia is is Canberra Canberra..";
    • every tool call fails as AI_InvalidToolInputError: each argument piece appears twice, so the JSON doesn't parse.

    A Worker counting the events of a short streamed answer through the binding found text in both fields in every text event, with or without tools:

    model events with text in both response and delta.content in one only
    llama-3.3-70b-instruct-fp8-fast 4 / 4 (11 / 11 with tools) 0
    llama-4-scout-17b-16e-instruct 7 / 7 (7 / 7 with tools) 0

    Tool-call events carried the same piece in tool_calls and choices[0].delta.tool_calls, every time.

  • After: three new tests in test/stream-text.test.ts (Binding - Streaming Text Tests). The first two fail on main and pass here:

    hybrid text events "The capital" + " is Canberra."   → "The capital is Canberra."   (main: "The capitalThe capital is Canberra. is Canberra.")
    hybrid tool-call pieces '{"location": ' '"Lon' 'don"}' → one get_weather call, input { location: "London" } (main: invalid input)
    native tool_calls with delta.tool_calls: []          → still one call (guards the new check)
    

    pnpm exec vitest --watch=false in packages/workers-ai-provider: 22 files, 457 tests pass (454 before, plus the 3 new). tsc --noEmit is clean.

Merge Danger

Door: two-way

Blast Radius: streaming

Only getMappedStream changes. Binding mode, credentials mode (REST /ai/run) and the AI Search chat model all read their streams through it. The one behaviour change, for an event that carries text in both fields: response now wins over delta.content. On Workers AI those were identical in every event observed.

@changeset-bot

changeset-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d2c054b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
workers-ai-provider Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant