Skip to content

ui: voice mode, a spoken conversation with the chat model - #65

Draft
allozaur wants to merge 10 commits into
allozaur/feat/voice-modefrom
allozaur/feat/voice-conversation
Draft

allozaur wants to merge 10 commits into
allozaur/feat/voice-modefrom
allozaur/feat/voice-conversation

Conversation

@allozaur

@allozaur allozaur commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Overview

Talk with the model hands-free: voice mode listens, sends what you say when you pause, and reads the reply aloud while it appears on screen, word by word. The chat form's main button on an empty message opens it; the microphone beside it keeps recording into the message.

Additional information

Part 4 of 13 of the work-in-progress stack on the fork, on top of #64 (voice messages and dictation), under Llama Harness (#55). The reasoning effort moves from the chat form's row into the model picker in the same PR, as the row was redesigned around the two voice buttons; the closed selector shows it next to the model when it is not the default.

Known gaps:

  • The full loop with a real microphone, and speaking over a reply to interrupt it, are not tested here: the browser used for testing blocks the microphone. The pieces are: transcription and reading aloud against LFM3-Omni-2B, the screen, and unit tests.
  • A turn ends after about 0.9 s of silence, judged in the browser from the input levels. Server-side turn detection (VAD) would do better, later.
  • Replies are shown and read as plain text in voice mode; markdown marks are left out of what is spoken, not rendered.
  • In router mode, a model that speaks other than the chat model is loaded to read replies; a server that keeps one model at a time (--models-max 1) swaps them every turn.
  • iOS: playback is unlocked by the click that opens voice mode; not tried on a device yet.
How it works
  • Listening: one VoiceRecorder per turn, with the browser's echo cancellation, noise suppression and auto gain on (unlike dictation, the reply plays from the same device). What is said shows live under the conversation, transcribed as in dictation: a live ASR model streams it, another model that hears gets it piece by piece at the pauses. The turn ends at a pause (PauseDetector, 0.9 s) and its text is sent with chatStore.sendMessage.
  • Reading aloud: ReplyReader follows the streaming reply, cuts complete sentences (readAloudPieces, list numbers are not sentence ends, markdown marks are dropped), speaks each through POST /v1/audio/speech and plays them in order, fetching the next one while the current one plays. Its spoken count follows playback, and VoiceSpokenText lights the reply up to it.
  • Which model speaks: the chat model when its architecture.output_modalities includes audio, else (router mode) the first model that does. None: replies are shown, not read, and the speaker button says why.
  • Moving through a reply: a click or tap on a word of the reply being read moves the voice there, back or ahead (ReplyReader.seek). A sentence already fetched plays at once, from about where the word falls in it, as the reply's speech is kept until the next reply.
  • Interrupting: while a reply plays, a level monitor listens; sound over it pauses the reading, and speech within a second stops it and the generation and starts a turn, else the reading goes on. Tapping the microphone does the same; muting it lets a held reply go on.
  • Composer: on an empty message the main button is voice mode (a waveform), the microphone a ghost button beside it. The reasoning effort is the last row of the model picker, in place of Manage models, opening its levels in place; a phone's model drawer has it too, and a single-model desktop selector, which has no picker, keeps its button. When the effort is a set level, the closed selector shows it as a word next to the model, and when reasoning is off a small crossed-out lightbulb; at the default it shows nothing more.
Testing
  • ui/apps/chat: npm run check (0 errors) and npm run lint pass, with 1035 unit, 373 client and 319 story tests.
  • New unit tests: readAloudPieces and toSpeech. New client tests: ReplyReader.seek (ahead, back into a sentence already read, nothing when muted or done) and VoiceSpokenText's word picks.
  • Against llama-server with LFM3-Omni-2B: /v1/models reports output_modalities: ["text", "audio"]; /v1/audio/speech read a sentence (3.2 s of audio in 0.6 s), which transcribed back to the same words. Voice mode opens from the composer and shows its greeting, status, microphone, model picker, speaker and close controls; the microphone itself was blocked in the test browser.
  • The reasoning row in the model picker, checked against a router server: it shows the level in use and opens Default, Off, Low, Medium, High and Max. The closed selector shows the word Low or Medium, a crossed-out lightbulb for Off, and nothing at Default.

Requirements

@allozaur
allozaur added this pull request to stack #62 November 16, 2026 09:43
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from cb3f315 to 8237d6e Compare November 16, 2026 10:15
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch 3 times, most recently from a287e12 to e77d0d3 Compare November 16, 2026 11:28
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from e77d0d3 to 0de3e3d Compare November 16, 2026 11:43
@allozaur
allozaur removed this pull request from stack #62 November 16, 2026 11:56
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 0de3e3d to 169b652 Compare November 16, 2026 11:56
@allozaur
allozaur added this pull request to stack #66 November 16, 2026 11:57
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch 2 times, most recently from 479efd8 to 51f0b08 Compare November 16, 2026 12:17
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 51f0b08 to 61d5e91 Compare November 16, 2026 13:09
@allozaur
allozaur removed this pull request from stack #66 November 16, 2026 13:44
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 61d5e91 to 8ce93c7 Compare November 16, 2026 13:44
@allozaur
allozaur added this pull request to stack #68 November 16, 2026 13:45
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 8ce93c7 to 75b53e3 Compare November 16, 2026 13:52
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 16, 2026
@allozaur
allozaur removed this pull request from stack #68 November 16, 2026 16:12
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 75b53e3 to 3362198 Compare November 16, 2026 16:12
@allozaur
allozaur added this pull request to stack #71 November 16, 2026 16:13
@allozaur
allozaur removed this pull request from stack #71 November 16, 2026 16:49
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch 2 times, most recently from 2e63e2c to 96786f6 Compare November 17, 2026 13:20
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 96786f6 to 56750d5 Compare November 17, 2026 13:44
@allozaur
allozaur removed this pull request from stack #76 November 17, 2026 13:44
@allozaur
allozaur added this pull request to stack #78 November 17, 2026 13:44
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from 56750d5 to e51e223 Compare November 17, 2026 15:28
@allozaur
allozaur removed this pull request from stack #78 November 17, 2026 16:25
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from e51e223 to e9b5386 Compare November 17, 2026 16:26
@allozaur
allozaur added this pull request to stack #79 November 17, 2026 16:26
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from e9b5386 to 6153f6c Compare November 17, 2026 16:42
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch 2 times, most recently from 5c70889 to bf6eb3c Compare November 17, 2026 17:20
@allozaur
allozaur force-pushed the allozaur/feat/voice-conversation branch from bf6eb3c to 8bffa7c Compare November 17, 2026 17:30
The chat form's main button on an empty message is now voice mode (a
waveform); the microphone steps back beside it and records as before.

Voice mode is a full-screen conversation:
- it listens with the browser's echo cancellation on, shows what is said
  as it is transcribed, and sends it after a pause of about a second;
- the reply streams on screen and is read aloud sentence by sentence
  through /v1/audio/speech, by the chat model when it speaks or else by
  another model that does (router mode), each word lit as it is spoken;
- speaking over the reply, or tapping the microphone, stops it and
  listens; the microphone and the speaker can be muted, and with no
  model that speaks the replies are shown without being read.

The voice engine gains synthesizeSpeech(), readAloudPieces() and
ReplyReader; models report output_modalities, so the app knows which
can speak.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 8c580d0)
The chat form's row no longer holds the reasoning button: the model
picker ends with a Reasoning row, in place of Manage models, that shows
the level in use and opens the levels under it. A phone's model drawer
has the same row. A desktop server with a single model, whose selector
opens the manager instead of a picker, keeps the button beside it.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit f491a9d)
- Settings > Voice > Reading speed: 1x to 2x, 1.25x by default, with
  the voice's pitch kept.
- The reply's first piece may end at a clause (a comma, colon or dash
  after 24 characters), so the voice starts before the first sentence
  is complete; later pieces stay whole sentences.
- The next two pieces are fetched while one plays.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit e71055f)
…een pieces

- A sound over a reply read aloud no longer stops it: the reading pauses
  and voice mode listens. Speech within a second, judged against the
  room's noise measured before, confirms the interruption; otherwise the
  reply goes on from where it paused. The reply's own echo or a noise
  used to stop it and leave the microphone listening. Tapping the
  microphone still interrupts outright.
- Read-aloud clips lose the silence the speech model puts around them
  (LFM3-Omni: about 140 ms before, 240 ms after), down to 40 ms and
  90 ms, so pieces, and paragraphs after an empty line, follow closely.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 653f357)
The voice mode screen no longer scrolls to the bottom as a reply
streams. A new exchange scrolls into view as it starts; then, while the
reply is read aloud, the page keeps the word being read in view, moving
only when it nears the bottom. With the speaker off, or no model that
speaks, it stays where it is.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit b64182c)
Voice mode shows no chat messages, so a reply that wanted to use a tool,
or an agentic loop asking to go on, waited on a card nobody could see.
Both cards now show above the microphone: they rise a little from below
as they fade in, and sink back as they fade out once answered (no motion
with reduced motion). Escape in the card's allow menu closes the menu,
not voice mode.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 91d1dc0)
An interruption, by voice or by tapping the microphone, now holds the
reply where it paused instead of stopping it. Muting the microphone, or
a turn in which nothing is understood, lets it go on from there; a turn
with words drops it, stopping its reading and generation, and is sent.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 704dd5f)
Space turns and the user's words from the reply further apart, most on
phones. A fade from the background over the bottom fifth of the screen
hides what is generated ahead of the reading, and the scroll keeps the
spoken word above it.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit e673c84)
While a reply is read aloud, a click or tap on one of its words moves
the voice there, back or ahead, and reading goes on from it. A sentence
already fetched plays at once, from about where the word falls in it:
the reply's speech is kept until the next one. A reply held for an
interruption goes on from the word too.

Assisted-by: Claude (Opus 5.5 Medium)
(cherry picked from commit 95d93e6)
…ctor

Next to the model on the closed selector, a set level shows as its word,
and reasoning that is off as a small crossed-out lightbulb. At the
default, which leaves it to the model, nothing more shows.

(cherry picked from commits 44b368f and f91f4a9 on develop)

Assisted-by: Claude (Opus 5.5 Medium)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation server/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant