Repository navigation
Conversation
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
2 times, most recently
from
November 16, 2026 07:05
0fdae5f to
f6aded4
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
2 times, most recently
from
November 16, 2026 09:24
4eaaab2 to
9092c09
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 10:42
6096cf2 to
f6cffe3
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 11:28
8a00774 to
0baf099
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 11:43
0baf099 to
1b83918
Compare
allozaur
changed the base branch from
allozaur/feat/prompt-library
to
allozaur/refactor/llama-harness-package
November 16, 2026 11:56
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 11:56
1b83918 to
417cd01
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 13:09
417cd01 to
7c5c805
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 13:44
7c5c805 to
232ae9a
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 13:52
232ae9a to
8c42efe
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 16, 2026 16:12
8c42efe to
598a0c3
Compare
allozaur
changed the base branch from
allozaur/refactor/llama-harness-package
to
allozaur/feat/llama-chat-npx
November 17, 2026 16:26
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 17, 2026 16:26
f832c37 to
4641ac1
Compare
This was referenced Nov 17, 2026
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 17, 2026 16:42
4641ac1 to
f123dd5
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
from
November 17, 2026 17:16
f123dd5 to
f6ebf9b
Compare
allozaur
force-pushed
the
allozaur/feat/voice-mode
branch
2 times, most recently
from
November 17, 2026 17:30
1d0ad6e to
d03c97f
Compare
Voice input needs a live waveform, a clip that can be sent and drawn, and a transcript, whether the model streams text while the user speaks or not. This adds the engine for it in src/lib/voice, with no imports of app stores or components, so it can move into a package later. - VoiceCapture: one microphone start gives live levels (RMS on a dB curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from an inline AudioWorklet (ScriptProcessor fallback), and on stop a 16-bit mono WAV clip with its duration and peaks. Tracks, nodes and the context are released on stop, cancel, a failed start and page hide. - computePeaks / computePeaksFromAudio: normalised peaks of samples, a buffer, or a stored base64 or Blob file. - transcribeClip: POST /v1/audio/transcriptions, with the server's own error message on failure. - RealtimeTranscriber: the /v1/realtime?intent=transcription socket, queueing audio until session.created, a final commit, and a typed RealtimeUnsupportedError when the model cannot stream or the socket is refused (a browser websocket cannot send the API key header). - startVoiceSession and VoiceRecorder: try realtime, fall back to one HTTP transcription of the clip, and expose levels and live text. Assisted-by: Claude (Opus 5.5 Medium)
Voice input needs three visuals that the app can feed with its own recording and transcription logic: the live waveform while recording, a playable waveform of a recorded clip, and a clip with its transcript that opens on hover, keyboard focus or a tap. VoiceWaveformLive draws input levels as many bars as fit the width, newest on the right, and holds still with reduced motion. VoiceWaveform plays a clip through its own audio element, fills the played part in the primary colour, seeks by press, drag and arrow keys (a slider) and pauses any other clip that was playing. VoiceTranscriptReveal keeps the transcript in the DOM for screen readers while it is closed, and shows transcribing and error states. The waveform math (resampling peaks, bar capacity, seek and time formatting) is in src/lib/voice with unit tests. The components are exported from the package root and from @ggml-org/llama-ui/voice, which Llama Chat resolves to the sources like the other entry points. Assisted-by: Claude (Opus 5.5 Medium)
Storybook gets UI/Blocks/Voice: the live waveform fed by a fake microphone meter in a chat form row, the clip waveform in its sizes and states, and the transcript reveal in each state and on touch. The demo audio is a WAV synthesised in the story, so no binary file is added. The play functions check seeking by keys and by press, the reveal on hover, keyboard focus and a tap, and a clip without audio. Pressing play needs a real click before a browser plays audio, so a client test covers play and pause and one clip pausing another. Assisted-by: Claude (Opus 5.5 Medium)
…that cannot hear A voice message recorded in the chat form is an audio attachment like an uploaded file, plus what the app needs to draw and read it. The audio attachment gets four optional fields: voice (recorded in the app), transcript, peaks and durationMs. Stored rows, export and import carry them as they are, so no migration is needed and older messages keep working. The transcript is for reading. A model with audio input still gets the audio alone; a model without it (the user switched models later) gets the transcript as a labelled text part in place of the audio. When the model's modalities are not known yet, or there is no transcript, the audio goes as before. A transcript often comes back after the message was sent. The new voiceTranscriptsStore follows each pending transcript by the clip's name, hands it to a send that happens after it arrived, and otherwise writes it into the stored attachment of the message it went out with, in the open chat or, if the user left it, in the database. Copying a message now includes its voice transcript. Assisted-by: Claude (Opus 5.5 Medium)
The user should not have to think about which kind of voice input is in use, so the app decides it from the conversation's model and two settings. resolveVoiceMode is a small pure function: with Voice input on Automatic and a chat model that has audio input, a recording is a voice message, transcribed by that same model so a router swaps nothing. Otherwise it is dictation, with the chat model when it can hear, else, in router mode, with the chosen transcription model. When no model can transcribe, the microphone is unavailable and says why. Settings get a Voice section of their own rather than more rows in General, which is already long: these four belong together and one of them lists the server's models. - Voice input: Automatic (default) or Always dictate. - Transcription model: Same as chat model (default) or, in router mode, any model with audio input; on a single-model server it only says whether that model can transcribe. It notes that dictating with another model loads it beside the chat model. - Transcribe voice messages: on by default. - Transcription language: empty for automatic. The keys live in the settings registry like the others, so defaults, storage, reset and export need nothing new. The model picker is drawn by SettingsChatVoiceTab, the way Extensions and Tools draw their sections. Assisted-by: Claude (Opus 5.5 Medium)
The microphone now does what the conversation's model allows, without the user choosing a mode: its label says "Send a voice message" or "Dictate", and when no model can take audio it is disabled and says why (shown in router mode only when some model there has audio input, so the reason can be acted on; a single-model server without audio keeps the send button as before). The microphone errors keep their warning state and toast, now also for the voice engine's own capture errors. Recording goes through the voice engine's VoiceRecorder instead of the MediaRecorder flow. While it records, the actions row becomes a recording row: cancel, the live waveform, the elapsed time and stop. Escape or cancel drops the recording. The form keeps its two rows while recording, so the message stays visible on desktop and the row fits the phone layouts and the expanded sheet. - Dictation writes the live transcript into the message at the caret as it streams. DictationSpan replaces only the dictated stretch, moves it when the user types before it, keeps what is typed after it, and hands text the user edits over to them. When nothing streams, the row shows Transcribing... after stop and the text goes in once it is back. Cancel takes the dictated text out again. No audio is attached. - A voice message streams its live text, if any, under the waveform, not into the message. On stop the clip is attached as a voice file with its peaks and length, and its transcript is followed by the voice transcripts store (unless Transcribe voice messages is off). Realtime transcription is skipped for servers reached through a tunnel or with an API key, which a browser websocket cannot carry. The first dictation with a separate router model shows a one-time note that it loads beside the chat model. The form takes an optional voiceRecorder, so tests and stories can run it without a microphone. Assisted-by: Claude (Opus 5.5 Medium)
A voice message should read as a clip, not as a file chip. Audio attachments now render with Llama UI's VoiceWaveform inside VoiceTranscriptReveal, through a direct branch in the core attachment item (voice is core, so it does not go through the extension renderers): the removable sm chip in the chat form, and the md clip in a sent message, on the user bubble's surface. - The audio plays from an object URL made from the stored base64 (or the uploaded file's data URL), revoked when the clip leaves the screen. - A clip stored without peaks, an older voice message or an uploaded audio file, gets them computed from its audio once per session, in memory only; uploaded files keep their name under the waveform. - The transcript opens under the clip on hover, keyboard focus or a tap, and shows Transcribing... while the voice transcripts store waits for it, Transcription unavailable when it failed, and nothing when none was asked for. - The full-screen preview keeps its player and shows the transcript. Tests run the chat form with a fake recorder: the microphone per mode, the recording row, dictation around typed text, the Transcribing... state, cancel, and a voice message chip whose transcript arrives later and goes with the send. Stories show the form recording each kind, a recorded chip, and sent voice messages with the transcript, pending, and computed peaks. Assisted-by: Claude (Opus 5.5 Medium)
The voice engine replaced AudioRecorder, so it goes. The dev server now proxies WebSockets on /v1, so realtime transcription works under vite dev. The VoiceCapture test waits for the tone to be recorded instead of a fixed 700 ms, which a loaded test run could miss. Assisted-by: Claude (Opus 5.5 Medium)
…annot hear The Transcription model setting's Automatic choice now falls back to a router model with audio input that is already loaded, so dictating with a text-only chat model works without picking a model and loads nothing. A picked model still wins, and the empty value is a named constant. Idea from upstream PR 27567, by NickM-27. Assisted-by: Claude (Opus 5.5 Medium)
- The live waveform fills the row from the start: a recording shorter than the row is stretched over all its bars, a longer one scrolls with the newest level on the right. No dotted placeholder. - A voice message's transcript is open under its clip, in the chat form while it comes in and in the sent message, with the toggle beside it to fold it. - Settings > Voice > Collapse transcripts (off by default) keeps them folded instead, opening on hover, keyboard focus or a tap. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 2a93f2f)
- Voice input asks for the raw microphone signal: no echo cancellation, noise suppression or auto gain. Echo cancellation made the system treat a recording as a call, which lowers or interrupts other playback, and auto gain faded the first words in. - One AudioContext serves every recording, with the capture worklet loaded once, suspended in between and, where the browser allows it, without an output device, so recording never touches the speakers. - The chat form prepares it when it gets focus (no microphone, so no recording indicator), and a mouse starts recording on press. - A recording starts counting once audio flows: until then the bar says Starting... instead of running the timer. Development builds log how long each step of a start took. - Settings > Voice > Microphone: Automatic (the system default, or the built-in microphone when the default is a Bluetooth headset, which would switch to its low-quality call mode), System default, or a listed microphone. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 76643e7)
…s spoken - A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text while you record: the realtime client commits the audio piece by piece, at the pauses the levels show, one piece at a time, and the final commit takes only what is left. - A voice message's live text shows as the empty message box's placeholder. Clicking into the box makes it text to edit there, and what is said next follows it. On stop the text leaves the box and goes with the clip; when it was edited, the edit is the clip's transcript. - Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the full foreground colour, which they keep, the rest muted. A clip's transcript follows its playback (estimated over the parts with speech, as there are no word timings), and live text brightens as it arrives. - The recording bar has a send button that stops and sends in one go. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit ce77301)
A capture starts with its first audio block, which can come before a stream's first sound, so the steady tone is checked from where it begins. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit a41ee53)
- A model that transcribes per commit (LFM3-Omni) gets shorter pieces: cut at a 0.35 s pause after speech, or at the next quiet moment once a piece reaches 3 s, so the text follows the speaker within a few seconds in dictation, the message box and voice mode. - When the recording stops, the audio after the last piece is committed only if speech was heard in it. LFM3-Omni transcribes a commit of pure silence as "Yeah." (or "はい。"), which ended up at the end of the text. A recording whose only piece is that rest still commits it. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 43e52cb)
Live text comes piece by piece, cut at the speaker's short pauses, and a model reads each piece as a sentence of its own: "I went. To the store." When the recording ends, one pass over the whole clip gives the final text, punctuated by meaning, in dictation and in voice mode alike. The pieces' text stays the live text until then, and is kept when that pass fails or the recording is longer than two minutes. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 359dd96)
A model that hears but cannot hold a realtime session, such as Qwen3-ASR, was transcribed only after the recording, so dictation showed no text while speaking. The recording is now cut at the speaker's pauses as with a realtime session, and each piece sent to /v1/audio/transcriptions on its own, one at a time, its text joined live. The audio is kept from the start, so what is said while the socket answers goes with the first piece. The whole clip is still transcribed once more at the end, and the pieces' text is kept if that fails. A server a websocket cannot reach (an API key, a remote server) gets live text the same way. Assisted-by: Claude (Opus 5.5 Medium)
…uest The browser keeps one processing setting per microphone: while a call, another tab or voice mode has it open with echo cancellation, a dictation that asked for the raw signal is processed too, sounds distant, and the system may lower other sound. The capture now warns when it gets processing it did not ask for, and its development log names the microphone, its rate and what processing is on. Assisted-by: Claude (Opus 5.5 Medium)
While a reply streams, or once the message has text, the microphone no longer gives way to the main button: it stays beside it as a quieter, secondary button, so more can be dictated without clearing the message or waiting for the reply. Assisted-by: Claude (Opus 5.5 Medium)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Talk to Llama Chat instead of typing. A model that can hear gets your voice as a playable voice message, with a transcript to read later; any other model gets your words as text, dictated into the message as you speak. The microphone picks the right one by itself.
Additional information
Part 3 of 13 of the work-in-progress stack on the fork, based on
allozaur/feat/llama-chat-npx(#69), the npx package, under voice mode (#65) and Llama Harness (#55). Its diff is only this layer. Since 17 November it sits below the harness, as it never used it; its commits are unchanged. Below the servers layer (#35), voice requests go to the local server; #35 sends them to the active one.Known gaps:
src/lib/voicetakes its dependencies as arguments, so it can later move into a package of its own (@ggml-org/llama-voice).Related upstream work:
ggml-org/llama.cpp#27567by NickM-27 (open, forggml-org/llama.cpp#24375) adds dictation with another loaded model and a transcription model setting. This layer covers the same ground and takes two things from it and its review: Automatic falls back to a model with audio input that is already loaded, and that choice is a named constant. As asked in that review, thelanguage English<asr_text>markup some ASR models (Qwen3-ASR) write is left for the server to strip; until it does, it shows up in their transcripts. The transcription model can also be one that isn't loaded, so dictation works with--models-max 1. Its picker is the library's Select, which the review notes has bits-ui's scroll jumps on long lists; the list holds only models with audio input. It builds on the server's/v1/audio/transcriptionsand/v1/realtime(ggml-org/llama.cpp#30636,ggml-org/llama.cpp#31269).Plan for upstream: this layer takes over
ggml-org/llama.cpp#27567rather than landing next to it, as both add a microphone for models that cannot hear. When it opens upstream, its description names ggml-org#27567 and NickM-27's work it builds on (the microphone for any chat model once a model with audio input is loaded, and the transcription model setting), and the dictation commit carriesCo-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>. ggml-org#27567 is then closed with a note pointing to the new PR; its wording is reviewed before it is posted.How the two kinds of voice input work
input_audio, as recordings are today. The same model transcribes it, so a router doesn't swap models. The transcript is stored with the clip for reading. It is never sent to a model that can hear; if a later turn goes to a model without audio input, the transcript goes as text in place of the audio./v1/realtime?intent=transcription) when the model has a streaming encoder and the server is reached directly without an API key, which WebSockets can't send; otherwise one request to/v1/audio/transcriptionsafter the recording. A model that hears but does not stream gets the speech piece by piece over the same socket, cut at short pauses, for live text; as each piece reads as a sentence of its own, the whole recording is transcribed once more when it ends (up to two minutes), so a pause inside a sentence does not end it.What changed
@ggml-org/llama-ui/voice):VoiceWaveformLivefor the recording,VoiceWaveformfor a playable clip, andVoiceTranscriptReveal, which shows the transcript below the clip on hover, keyboard focus or tap. Presentational only, with stories and tests.src/lib/voice): microphone capture with live levels and PCM, WAV encoding, waveform peaks, the realtime and HTTP transcription clients and aVoiceRecorderthat ties them together.voice,transcript,peaksanddurationMs. No migration: the fields aren't indexed, and export and import carry them. A transcript that arrives after its message was sent is written into the stored message.AudioRecorderis removed, and the dev server proxies WebSockets on/v1so realtime transcription works undervite dev.Testing
llama-serverwith LFM3-Omni-2B (audio input), on desktop and phone widths in light and dark: a recorded voice message was answered from the audio ("Paris") and got its transcript, stored with the clip in IndexedDB; the same sentence dictated came in as text with no attachment. The microphone was a stream playing a spoken WAV, since Chromium's fake device crashes the page in this environment; everything after the device is the real app.ui/apps/chat:npm run check(0 errors, the 2 existing warnings),npm run lintand the production build pass, with 981 unit tests, 361 client tests (three full runs) and all 316 story tests.ui/lib/ui:npm run checkand its 65 tests pass.Requirements
Claude (Opus 5.5 Medium)