Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f8486a9
ui: add a voice engine for capture, peaks and transcription
allozaur Nov 15, 2026
a7c2576
ui: add voice waveform and transcript components to Llama UI
allozaur Nov 15, 2026
6b712ca
ui: add stories and tests for the voice components
allozaur Nov 15, 2026
a62b902
ui: store voice messages with their transcript and send it to models …
allozaur Nov 15, 2026
a93442f
ui: add Voice settings and resolve what the microphone does per model
allozaur Nov 15, 2026
8bb7b9d
ui: record voice messages and dictation from the chat form
allozaur Nov 15, 2026
e0412fe
ui: draw audio attachments as playable waveforms with their transcript
allozaur Nov 15, 2026
eb2fc37
ui: drop the old audio recorder and proxy realtime transcription in dev
allozaur Nov 15, 2026
4364aa9
ui: dictate with a model that is already loaded when the chat model c…
allozaur Nov 16, 2026
b9e1120
ui: fill the recording waveform, and open voice message transcripts
allozaur Nov 16, 2026
7a56f5c
ui: record voice without ducking other audio, from the first word
allozaur Nov 16, 2026
f74c3fc
ui: show voice text live in the message box, and follow transcripts a…
allozaur Nov 16, 2026
bfc1722
ui: let the capture test start with silence
allozaur Nov 16, 2026
a87c466
ui: show live voice text sooner, and keep silence from becoming "yeah"
allozaur Nov 16, 2026
ef4b89b
ui: transcribe a recording cut at pauses once more as a whole
allozaur Nov 16, 2026
f588748
ui: give live text over HTTP for models without a realtime session
allozaur Nov 17, 2026
7d1a07c
ui: say when the browser processes the microphone despite the raw req…
allozaur Nov 17, 2026
c3efab2
ui: keep the microphone beside the stop and send buttons
allozaur Nov 17, 2026
12352ee
ui: show the voice components' stories with Llama UI
allozaur Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ui/apps/chat/.storybook/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ const config: StorybookConfig = {
'../tests/stories/**/*.stories.@(js|ts|svelte)',
{
directory: '../../../lib/ui/tests/stories',
files: '{foundations,components,app,models,mockups}/**/*.stories.svelte'
files: '{foundations,components,app,models,mockups,voice}/**/*.stories.svelte'
}
],
addons: [
Expand Down
6 changes: 6 additions & 0 deletions ui/apps/chat/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1151,6 +1151,12 @@ icon buttons in the chat form are fully round.
(`--sidebar-reveal-x`), taking no taps while the page is aside. The page keeps room under its
last row (`max-md:pb-safe-24`) so the key never covers it. One per page, only for its main
action; from `md` up the action stays a button in the header.
- **Voice clips:** a recorded clip is a round primary play key, its waveform and the time in
`caption`: bars in the muted foreground at 40%, the played part in the primary colour, so
the clip stays neutral in both modes. The live waveform while recording is as tall as the
input's line (24px), the newest level on the right, and holds still with reduced motion. A
voice message's transcript opens under the clip on hover, keyboard focus or a tap, and
stays in the DOM for screen readers while closed.
- **Copy feedback:** a copy action turns its icon into a `fg-success` check for 1.5s
(`COPY_FEEDBACK_RESET_MS`), only when the copy succeeded, then back. The icon pops in with a
150ms fade and scale, skipped on first render and with reduced motion.
Expand Down
1 change: 1 addition & 0 deletions ui/apps/chat/scripts/llama-ui.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ const LLAMA_UI_EXPORTS = [
['@ggml-org/llama-ui/enums', `${source}/enums/index.ts`],
['@ggml-org/llama-ui/code', `${source}/utils/index.ts`],
['@ggml-org/llama-ui/utils', `${source}/components/ui/utils.ts`],
['@ggml-org/llama-ui/voice', `${source}/components/voice/index.ts`],
['@ggml-org/llama-ui/styles/*', `${source}/styles/*`],
['@ggml-org/llama-ui/theme.css', `${source}/styles/theme.css`],
['@ggml-org/llama-ui/tokens.css', `${source}/styles/tokens.css`],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
ChatAttachmentsListItemMcpPrompt,
ChatAttachmentsListItemMcpResource,
ChatAttachmentsListItemThumbnailFile,
ChatAttachmentsListItemThumbnailImage
ChatAttachmentsListItemThumbnailImage,
ChatAttachmentsListItemVoice
} from '#lib/components/app/index.ts';
import { AttachmentType } from '#lib/enums/index.ts';
import type {
Expand All @@ -12,7 +13,7 @@
DatabaseMessageExtraMcpResource,
MCPResourceAttachment
} from '#lib/types/index.ts';
import { isMcpPrompt, isMcpResource, isPdfFile } from '#lib/utils/index.ts';
import { isAudioFile, isMcpPrompt, isMcpResource, isPdfFile } from '#lib/utils/index.ts';

interface Props {
class?: string;
Expand Down Expand Up @@ -93,6 +94,15 @@
class="flex-shrink-0 {className} {scrollClasses}"
onclick={() => onMcpResourcePreview?.(mcpResource)}
/>
{:else if isAudioFile(item.attachment, item.uploadedFile)}
<!-- a clip plays where it is: the waveform takes the presses a chip would -->
<ChatAttachmentsListItemVoice
class="{layoutClasses} {className} {scrollClasses}"
{item}
onRemove={onFileRemove}
{readonly}
size={readonly ? 'md' : 'sm'}
/>
{:else if item.isImage && item.preview}
<ChatAttachmentsListItemThumbnailImage
class="{layoutClasses} cursor-pointer {className} {scrollClasses}"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
<!--
An audio attachment drawn as a clip: a playable waveform, and for a voice message its transcript under it, open while
it comes in and after, unless the Collapse transcripts setting folds it (then it opens on hover, keyboard focus or a
tap). `sm` is the chip in the chat form, `md` the clip in a sent message. A clip
stored without peaks (an older message, an uploaded file) gets them computed from its audio.
-->
<script lang="ts">
import {
spokenLength,
VoiceTranscriptReveal,
type VoiceTranscriptStatus,
VoiceWaveform
} from '@ggml-org/llama-ui/voice';
import { X } from '@lucide/svelte';
import { ActionIcon } from '#lib/components/app/index.ts';
import { SETTINGS_KEYS } from '#lib/constants/index.ts';
import { settingsStore, voiceTranscriptsStore } from '#lib/stores/index.ts';
import type { ChatAttachmentDisplayItem } from '#lib/types/index.ts';
import { audioUrl, lazyPeaks, voiceAttachmentSource } from '#lib/utils/voice-attachment.ts';

interface Props {
class?: string;
item: ChatAttachmentDisplayItem;
onRemove?: (id: string) => void;
readonly?: boolean;
size?: 'sm' | 'md';
}

let { class: className = '', item, onRemove, readonly = false, size = 'md' }: Props = $props();

let source = $derived(voiceAttachmentSource(item));
let computedPeaks = $state<number[] | null>(null);
let src = $state<string | undefined>();

let peaks = $derived(source.peaks ?? computedPeaks ?? []);
let pending = $derived(voiceTranscriptsStore.get(source.name));
let transcript = $derived(
source.transcript ?? (pending?.status === 'done' ? pending.transcript : undefined)
);
let status = $derived<VoiceTranscriptStatus>(
transcript !== undefined
? 'done'
: pending?.status === 'transcribing'
? 'transcribing'
: pending?.status === 'error'
? 'error'
: 'idle'
);
let label = $derived(source.voice ? 'Voice message' : source.name);
// the transcript lights up word by word as the clip plays, and stays lit
let playbackMs = $state<number | null>(null);
let spoken = $derived(
transcript && playbackMs !== null && source.durationMs
? spokenLength(transcript, peaks, source.durationMs, playbackMs)
: null
);
let collapsed = $derived(settingsStore.config[SETTINGS_KEYS.VOICE_COLLAPSE_TRANSCRIPTS] === true);

// an object URL lives as long as the clip is on screen
$effect(() => {
const audio = source.audio;

if (!audio) {
src = undefined;

return;
}

const playable = audioUrl(audio, source.mimeType);

src = playable?.url;

return () => {
if (playable?.revoke) URL.revokeObjectURL(playable.url);
};
});

$effect(() => {
const { audio, name, peaks: stored } = source;

computedPeaks = null;

if (stored || !audio) return;

let cancelled = false;

lazyPeaks(name, audio).then((value) => {
if (!cancelled) computedPeaks = value;
});

return () => {
cancelled = true;
};
});
</script>

<div
class={[
'group relative min-w-0',
size === 'sm'
? 'w-60 max-w-[calc(100vw-4rem)] rounded-lg border border-border bg-muted py-2 pr-9 pl-2'
: 'w-80 max-w-full rounded-[1.125rem] matte-sheen bg-(--user-bubble-background) px-3 py-2.5 backdrop-blur-md dark:bg-primary/15',
className
]}
data-slot="chat-attachment-voice"
data-voice={source.voice || undefined}
>
{#key collapsed}
<VoiceTranscriptReveal defaultOpen={!collapsed} {spoken} {status} {transcript}>
<VoiceWaveform bind:playbackMs durationMs={source.durationMs} {label} {peaks} {size} {src} />
</VoiceTranscriptReveal>
{/key}

{#if !source.voice}
<p class="mt-1 truncate px-1 text-caption text-fg-muted">{source.name}</p>
{/if}

{#if !readonly && onRemove}
<div class="absolute top-1/2 right-1 -translate-y-1/2">
<ActionIcon
ariaLabel={source.voice ? 'Remove voice message' : `Remove ${source.name}`}
icon={X}
onclick={() => onRemove?.(item.id)}
stopPropagationOnClick
tooltip="Remove"
/>
</div>
{/if}
</div>
Original file line number Diff line number Diff line change
@@ -1,12 +1,24 @@
<script lang="ts">
import { Music } from '@lucide/svelte';
import { voiceTranscriptsStore } from '#lib/stores/index.ts';
import type { ChatAttachmentDisplayItem } from '#lib/types/index.ts';
import { voiceAttachmentSource } from '#lib/utils/voice-attachment.ts';

interface Props {
currentItem: { name?: string } | null;
currentItem: ChatAttachmentDisplayItem | null;
audioSrc: string | null;
}

let { audioSrc, currentItem }: Props = $props();

let source = $derived(currentItem ? voiceAttachmentSource(currentItem) : null);
// a voice message's transcript, from the message or still on its way
let transcript = $derived(
source ? (source.transcript ?? voiceTranscriptsStore.transcriptOf(source.name)) : undefined
);
let transcribing = $derived(
!!source && !transcript && voiceTranscriptsStore.get(source.name)?.status === 'transcribing'
);
</script>

<div class="flex flex-1 items-center justify-center p-8">
Expand All @@ -21,6 +33,19 @@
<p class="mb-4 text-white/70">Audio preview not available</p>
{/if}

<p class="text-sm text-white/50">{currentItem?.name || 'Audio'}</p>
<p class="text-sm text-white/50">
{source?.voice ? 'Voice message' : currentItem?.name || 'Audio'}
</p>

{#if transcript}
<p
class="mt-6 max-h-[40vh] overflow-y-auto text-left text-body-sm whitespace-pre-wrap text-white/80"
data-slot="voice-preview-transcript"
>
<span class="sr-only">Transcript: </span>{transcript}
</p>
{:else if transcribing}
<p class="mt-6 text-caption text-white/50">Transcribing...</p>
{/if}
</div>
</div>
Loading
Loading