Skip to content

ui: voice messages and dictation with live waveforms and transcripts - #64

Draft
allozaur wants to merge 18 commits into
allozaur/feat/llama-chat-npxfrom
allozaur/feat/voice-mode
Draft

allozaur wants to merge 18 commits into
allozaur/feat/llama-chat-npxfrom
allozaur/feat/voice-mode

Conversation

@allozaur

@allozaur allozaur commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Overview

Talk to Llama Chat instead of typing. A model that can hear gets your voice as a playable voice message, with a transcript to read later; any other model gets your words as text, dictated into the message as you speak. The microphone picks the right one by itself.

Additional information

Part 3 of 13 of the work-in-progress stack on the fork, based on allozaur/feat/llama-chat-npx (#69), the npx package, under voice mode (#65) and Llama Harness (#55). Its diff is only this layer. Since 17 November it sits below the harness, as it never used it; its commits are unchanged. Below the servers layer (#35), voice requests go to the local server; #35 sends them to the active one.

Known gaps:

  • A transcript still being written when the page reloads is lost; the clip then shows without one.
  • Live streaming dictation (realtime transcription) was tested only with a fake recorder. The real model used here doesn't stream, so the app fell back to one HTTP transcription after the recording, as designed.
  • Router mode, the transcription model picker and sending a transcript to a model without audio are covered by unit tests only.
  • The voice engine in src/lib/voice takes its dependencies as arguments, so it can later move into a package of its own (@ggml-org/llama-voice).

Related upstream work: ggml-org/llama.cpp#27567 by NickM-27 (open, for ggml-org/llama.cpp#24375) adds dictation with another loaded model and a transcription model setting. This layer covers the same ground and takes two things from it and its review: Automatic falls back to a model with audio input that is already loaded, and that choice is a named constant. As asked in that review, the language English<asr_text> markup some ASR models (Qwen3-ASR) write is left for the server to strip; until it does, it shows up in their transcripts. The transcription model can also be one that isn't loaded, so dictation works with --models-max 1. Its picker is the library's Select, which the review notes has bits-ui's scroll jumps on long lists; the list holds only models with audio input. It builds on the server's /v1/audio/transcriptions and /v1/realtime (ggml-org/llama.cpp#30636, ggml-org/llama.cpp#31269).

Plan for upstream: this layer takes over ggml-org/llama.cpp#27567 rather than landing next to it, as both add a microphone for models that cannot hear. When it opens upstream, its description names ggml-org#27567 and NickM-27's work it builds on (the microphone for any chat model once a model with audio input is loaded, and the transcription model setting), and the dictation commit carries Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>. ggml-org#27567 is then closed with a note pointing to the new PR; its wording is reviewed before it is posted.

How the two kinds of voice input work
  • Voice message: when the conversation's model has audio input (an omni model), the recording is attached as a WAV clip and sent as input_audio, as recordings are today. The same model transcribes it, so a router doesn't swap models. The transcript is stored with the clip for reading. It is never sent to a model that can hear; if a later turn goes to a model without audio input, the transcript goes as text in place of the audio.
  • Dictation: when the model can't hear, or Voice input is set to Always dictate, a model that can transcribe turns the speech into text in the message, at the caret, while you speak. In router mode that can be any model with audio input (Settings > Voice > Transcription model); a single-model server without audio input has no dictation, and the microphone says why. No audio reaches the conversation.
  • Transcription: over the realtime WebSocket (/v1/realtime?intent=transcription) when the model has a streaming encoder and the server is reached directly without an API key, which WebSockets can't send; otherwise one request to /v1/audio/transcriptions after the recording. A model that hears but does not stream gets the speech piece by piece over the same socket, cut at short pauses, for live text; as each piece reads as a sentence of its own, the whole recording is transcribed once more when it ends (up to two minutes), so a pause inside a sentence does not end it.
What changed
  • Llama UI (@ggml-org/llama-ui/voice): VoiceWaveformLive for the recording, VoiceWaveform for a playable clip, and VoiceTranscriptReveal, which shows the transcript below the clip on hover, keyboard focus or tap. Presentational only, with stories and tests.
  • Voice engine (src/lib/voice): microphone capture with live levels and PCM, WAV encoding, waveform peaks, the realtime and HTTP transcription clients and a VoiceRecorder that ties them together.
  • Data: audio attachments gain optional voice, transcript, peaks and durationMs. No migration: the fields aren't indexed, and export and import carry them. A transcript that arrives after its message was sent is written into the stored message.
  • Settings: a Voice section with Voice input (Automatic or Always dictate), Transcription model (router mode: Automatic, the chat model when it hears and else a loaded model with audio input, or one you pick), Transcribe voice messages (on by default) and Language.
  • Chat form: on an empty message the microphone takes the send button's place; once the message has text, or while a reply streams, it stays as a quieter button beside send or stop, so more can be dictated at any time. While recording, the actions row shows cancel, the live waveform, the time and stop; Escape cancels. Dictated text streams into the message and only the dictated span is replaced, so typing around it is kept. A voice message becomes a small playable chip in the form.
  • Messages: every audio attachment, including uploaded files, is drawn as a playable waveform, with the transcript revealed below it; the full-screen preview shows the transcript too. Copying a message includes its transcripts.
  • The old AudioRecorder is removed, and the dev server proxies WebSockets on /v1 so realtime transcription works under vite dev.
Testing
  • End to end against a local llama-server with LFM3-Omni-2B (audio input), on desktop and phone widths in light and dark: a recorded voice message was answered from the audio ("Paris") and got its transcript, stored with the clip in IndexedDB; the same sentence dictated came in as text with no attachment. The microphone was a stream playing a spoken WAV, since Chromium's fake device crashes the page in this environment; everything after the device is the real app.
  • ui/apps/chat: npm run check (0 errors, the 2 existing warnings), npm run lint and the production build pass, with 981 unit tests, 361 client tests (three full runs) and all 316 story tests.
  • ui/lib/ui: npm run check and its 65 tests pass.
  • New tests: mode resolution and its settings, what a voice message sends to models with and without audio input, the dictated span, the chat form's recording flow with a fake recorder (dictation around typed text, cancel, a transcript arriving after the send), capture in a real browser fed by a synthesized microphone, and the waveform and transcript components. Stories for the recording states, voice messages and the Voice settings.

Requirements

@allozaur
allozaur added this pull request to stack #62 November 16, 2026 05:34
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 16, 2026
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch 2 times, most recently from 0fdae5f to f6aded4 Compare November 16, 2026 07:05
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch 2 times, most recently from 4eaaab2 to 9092c09 Compare November 16, 2026 09:24
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 6096cf2 to f6cffe3 Compare November 16, 2026 10:42
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 8a00774 to 0baf099 Compare November 16, 2026 11:28
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 0baf099 to 1b83918 Compare November 16, 2026 11:43
@allozaur
allozaur removed this pull request from stack #62 November 16, 2026 11:56
@allozaur
allozaur changed the base branch from allozaur/feat/prompt-library to allozaur/refactor/llama-harness-package November 16, 2026 11:56
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 1b83918 to 417cd01 Compare November 16, 2026 11:56
@allozaur
allozaur added this pull request to stack #66 November 16, 2026 11:57
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 417cd01 to 7c5c805 Compare November 16, 2026 13:09
@allozaur
allozaur removed this pull request from stack #66 November 16, 2026 13:44
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 7c5c805 to 232ae9a Compare November 16, 2026 13:44
@allozaur
allozaur added this pull request to stack #68 November 16, 2026 13:45
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 232ae9a to 8c42efe Compare November 16, 2026 13:52
@allozaur
allozaur removed this pull request from stack #68 November 16, 2026 16:12
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 8c42efe to 598a0c3 Compare November 16, 2026 16:12
@allozaur
allozaur added this pull request to stack #71 November 16, 2026 16:13
@allozaur
allozaur removed this pull request from stack #71 November 16, 2026 16:49
@allozaur
allozaur removed this pull request from stack #78 November 17, 2026 16:25
@allozaur
allozaur changed the base branch from allozaur/refactor/llama-harness-package to allozaur/feat/llama-chat-npx November 17, 2026 16:26
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from f832c37 to 4641ac1 Compare November 17, 2026 16:26
@allozaur
allozaur added this pull request to stack #79 November 17, 2026 16:26
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from 4641ac1 to f123dd5 Compare November 17, 2026 16:42
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch from f123dd5 to f6ebf9b Compare November 17, 2026 17:16
@allozaur
allozaur force-pushed the allozaur/feat/voice-mode branch 2 times, most recently from 1d0ad6e to d03c97f Compare November 17, 2026 17:30
Voice input needs a live waveform, a clip that can be sent and drawn,
and a transcript, whether the model streams text while the user speaks
or not. This adds the engine for it in src/lib/voice, with no imports
of app stores or components, so it can move into a package later.

- VoiceCapture: one microphone start gives live levels (RMS on a dB
  curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from
  an inline AudioWorklet (ScriptProcessor fallback), and on stop a
  16-bit mono WAV clip with its duration and peaks. Tracks, nodes and
  the context are released on stop, cancel, a failed start and page hide.
- computePeaks / computePeaksFromAudio: normalised peaks of samples, a
  buffer, or a stored base64 or Blob file.
- transcribeClip: POST /v1/audio/transcriptions, with the server's own
  error message on failure.
- RealtimeTranscriber: the /v1/realtime?intent=transcription socket,
  queueing audio until session.created, a final commit, and a typed
  RealtimeUnsupportedError when the model cannot stream or the socket
  is refused (a browser websocket cannot send the API key header).
- startVoiceSession and VoiceRecorder: try realtime, fall back to one
  HTTP transcription of the clip, and expose levels and live text.

Assisted-by: Claude (Opus 5.5 Medium)
Voice input needs three visuals that the app can feed with its own
recording and transcription logic: the live waveform while recording,
a playable waveform of a recorded clip, and a clip with its transcript
that opens on hover, keyboard focus or a tap.

VoiceWaveformLive draws input levels as many bars as fit the width,
newest on the right, and holds still with reduced motion.
VoiceWaveform plays a clip through its own audio element, fills the
played part in the primary colour, seeks by press, drag and arrow keys
(a slider) and pauses any other clip that was playing.
VoiceTranscriptReveal keeps the transcript in the DOM for screen
readers while it is closed, and shows transcribing and error states.

The waveform math (resampling peaks, bar capacity, seek and time
formatting) is in src/lib/voice with unit tests. The components are
exported from the package root and from @ggml-org/llama-ui/voice, which
Llama Chat resolves to the sources like the other entry points.

Assisted-by: Claude (Opus 5.5 Medium)
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake
microphone meter in a chat form row, the clip waveform in its sizes and
states, and the transcript reveal in each state and on touch. The demo
audio is a WAV synthesised in the story, so no binary file is added.

The play functions check seeking by keys and by press, the reveal on
hover, keyboard focus and a tap, and a clip without audio. Pressing play
needs a real click before a browser plays audio, so a client test covers
play and pause and one clip pausing another.

Assisted-by: Claude (Opus 5.5 Medium)
…that cannot hear

A voice message recorded in the chat form is an audio attachment like an
uploaded file, plus what the app needs to draw and read it. The audio
attachment gets four optional fields: voice (recorded in the app),
transcript, peaks and durationMs. Stored rows, export and import carry
them as they are, so no migration is needed and older messages keep
working.

The transcript is for reading. A model with audio input still gets the
audio alone; a model without it (the user switched models later) gets the
transcript as a labelled text part in place of the audio. When the
model's modalities are not known yet, or there is no transcript, the
audio goes as before.

A transcript often comes back after the message was sent. The new
voiceTranscriptsStore follows each pending transcript by the clip's name,
hands it to a send that happens after it arrived, and otherwise writes it
into the stored attachment of the message it went out with, in the open
chat or, if the user left it, in the database.

Copying a message now includes its voice transcript.

Assisted-by: Claude (Opus 5.5 Medium)
The user should not have to think about which kind of voice input is in
use, so the app decides it from the conversation's model and two
settings. resolveVoiceMode is a small pure function: with Voice input on
Automatic and a chat model that has audio input, a recording is a voice
message, transcribed by that same model so a router swaps nothing.
Otherwise it is dictation, with the chat model when it can hear, else,
in router mode, with the chosen transcription model. When no model can
transcribe, the microphone is unavailable and says why.

Settings get a Voice section of their own rather than more rows in
General, which is already long: these four belong together and one of
them lists the server's models.

- Voice input: Automatic (default) or Always dictate.
- Transcription model: Same as chat model (default) or, in router mode,
  any model with audio input; on a single-model server it only says
  whether that model can transcribe. It notes that dictating with
  another model loads it beside the chat model.
- Transcribe voice messages: on by default.
- Transcription language: empty for automatic.

The keys live in the settings registry like the others, so defaults,
storage, reset and export need nothing new. The model picker is drawn by
SettingsChatVoiceTab, the way Extensions and Tools draw their sections.

Assisted-by: Claude (Opus 5.5 Medium)
The microphone now does what the conversation's model allows, without
the user choosing a mode: its label says "Send a voice message" or
"Dictate", and when no model can take audio it is disabled and says why
(shown in router mode only when some model there has audio input, so the
reason can be acted on; a single-model server without audio keeps the
send button as before). The microphone errors keep their warning state
and toast, now also for the voice engine's own capture errors.

Recording goes through the voice engine's VoiceRecorder instead of the
MediaRecorder flow. While it records, the actions row becomes a
recording row: cancel, the live waveform, the elapsed time and stop.
Escape or cancel drops the recording. The form keeps its two rows while
recording, so the message stays visible on desktop and the row fits the
phone layouts and the expanded sheet.

- Dictation writes the live transcript into the message at the caret
  as it streams. DictationSpan replaces only the dictated stretch, moves
  it when the user types before it, keeps what is typed after it, and
  hands text the user edits over to them. When nothing streams, the row
  shows Transcribing... after stop and the text goes in once it is back.
  Cancel takes the dictated text out again. No audio is attached.
- A voice message streams its live text, if any, under the waveform,
  not into the message. On stop the clip is attached as a voice file
  with its peaks and length, and its transcript is followed by the voice
  transcripts store (unless Transcribe voice messages is off).

Realtime transcription is skipped for servers reached through a tunnel
or with an API key, which a browser websocket cannot carry. The first
dictation with a separate router model shows a one-time note that it
loads beside the chat model.

The form takes an optional voiceRecorder, so tests and stories can run
it without a microphone.

Assisted-by: Claude (Opus 5.5 Medium)
A voice message should read as a clip, not as a file chip. Audio
attachments now render with Llama UI's VoiceWaveform inside
VoiceTranscriptReveal, through a direct branch in the core attachment
item (voice is core, so it does not go through the extension renderers):
the removable sm chip in the chat form, and the md clip in a sent
message, on the user bubble's surface.

- The audio plays from an object URL made from the stored base64 (or the
  uploaded file's data URL), revoked when the clip leaves the screen.
- A clip stored without peaks, an older voice message or an uploaded
  audio file, gets them computed from its audio once per session, in
  memory only; uploaded files keep their name under the waveform.
- The transcript opens under the clip on hover, keyboard focus or a tap,
  and shows Transcribing... while the voice transcripts store waits for
  it, Transcription unavailable when it failed, and nothing when none
  was asked for.
- The full-screen preview keeps its player and shows the transcript.

Tests run the chat form with a fake recorder: the microphone per mode,
the recording row, dictation around typed text, the Transcribing...
state, cancel, and a voice message chip whose transcript arrives later
and goes with the send. Stories show the form recording each kind, a
recorded chip, and sent voice messages with the transcript, pending,
and computed peaks.

Assisted-by: Claude (Opus 5.5 Medium)
The voice engine replaced AudioRecorder, so it goes. The dev server now
proxies WebSockets on /v1, so realtime transcription works under vite dev.
The VoiceCapture test waits for the tone to be recorded instead of a fixed
700 ms, which a loaded test run could miss.

Assisted-by: Claude (Opus 5.5 Medium)
…annot hear

The Transcription model setting's Automatic choice now falls back to a
router model with audio input that is already loaded, so dictating with a
text-only chat model works without picking a model and loads nothing.
A picked model still wins, and the empty value is a named constant.
Idea from upstream PR 27567, by NickM-27.

Assisted-by: Claude (Opus 5.5 Medium)
- The live waveform fills the row from the start: a recording shorter
  than the row is stretched over all its bars, a longer one scrolls with
  the newest level on the right. No dotted placeholder.
- A voice message's transcript is open under its clip, in the chat form
  while it comes in and in the sent message, with the toggle beside it
  to fold it.
- Settings > Voice > Collapse transcripts (off by default) keeps them
  folded instead, opening on hover, keyboard focus or a tap.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 2a93f2f)
- Voice input asks for the raw microphone signal: no echo cancellation,
  noise suppression or auto gain. Echo cancellation made the system treat
  a recording as a call, which lowers or interrupts other playback, and
  auto gain faded the first words in.
- One AudioContext serves every recording, with the capture worklet
  loaded once, suspended in between and, where the browser allows it,
  without an output device, so recording never touches the speakers.
- The chat form prepares it when it gets focus (no microphone, so no
  recording indicator), and a mouse starts recording on press.
- A recording starts counting once audio flows: until then the bar says
  Starting... instead of running the timer. Development builds log how
  long each step of a start took.
- Settings > Voice > Microphone: Automatic (the system default, or the
  built-in microphone when the default is a Bluetooth headset, which
  would switch to its low-quality call mode), System default, or a
  listed microphone.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 76643e7)
…s spoken

- A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text
  while you record: the realtime client commits the audio piece by piece,
  at the pauses the levels show, one piece at a time, and the final
  commit takes only what is left.
- A voice message's live text shows as the empty message box's
  placeholder. Clicking into the box makes it text to edit there, and
  what is said next follows it. On stop the text leaves the box and goes
  with the clip; when it was edited, the edit is the clip's transcript.
- Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the
  full foreground colour, which they keep, the rest muted. A clip's
  transcript follows its playback (estimated over the parts with speech,
  as there are no word timings), and live text brightens as it arrives.
- The recording bar has a send button that stops and sends in one go.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit ce77301)
A capture starts with its first audio block, which can come before a
stream's first sound, so the steady tone is checked from where it begins.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit a41ee53)
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces:
  cut at a 0.35 s pause after speech, or at the next quiet moment once a
  piece reaches 3 s, so the text follows the speaker within a few
  seconds in dictation, the message box and voice mode.
- When the recording stops, the audio after the last piece is committed
  only if speech was heard in it. LFM3-Omni transcribes a commit of pure
  silence as "Yeah." (or "はい。"), which ended up at the end of the text.
  A recording whose only piece is that rest still commits it.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 43e52cb)
Live text comes piece by piece, cut at the speaker's short pauses, and a
model reads each piece as a sentence of its own: "I went. To the store."
When the recording ends, one pass over the whole clip gives the final
text, punctuated by meaning, in dictation and in voice mode alike. The
pieces' text stays the live text until then, and is kept when that pass
fails or the recording is longer than two minutes.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 359dd96)
A model that hears but cannot hold a realtime session, such as
Qwen3-ASR, was transcribed only after the recording, so dictation
showed no text while speaking. The recording is now cut at the
speaker's pauses as with a realtime session, and each piece sent to
/v1/audio/transcriptions on its own, one at a time, its text joined
live. The audio is kept from the start, so what is said while the
socket answers goes with the first piece. The whole clip is still
transcribed once more at the end, and the pieces' text is kept if
that fails. A server a websocket cannot reach (an API key, a remote
server) gets live text the same way.

Assisted-by: Claude (Opus 5.5 Medium)
…uest

The browser keeps one processing setting per microphone: while a call,
another tab or voice mode has it open with echo cancellation, a
dictation that asked for the raw signal is processed too, sounds
distant, and the system may lower other sound. The capture now warns
when it gets processing it did not ask for, and its development log
names the microphone, its rate and what processing is on.

Assisted-by: Claude (Opus 5.5 Medium)
While a reply streams, or once the message has text, the microphone no longer
gives way to the main button: it stays beside it as a quieter, secondary
button, so more can be dictated without clearing the message or waiting for
the reply.

Assisted-by: Claude (Opus 5.5 Medium)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation server/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant