Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 193 additions & 0 deletions skills/ui-demo/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
---
name: ui-demo
description: Demo, test and document a change in Llama Chat (ui/apps/chat) or Llama UI (ui/lib/ui) with the Playwright demo harness - record a video for the PR, turn the same flow into an e2e check, and capture docs videos and screenshots. Use when a UI change needs a demo video, a Playwright check, or docs media.
---

# Demo, test and document a UI change

One scripted flow serves three ends: a **demo** video in the PR, a Playwright **check** in `tests/e2e`, and **docs** media. The harness is `ui/apps/chat/demo/` (read its `README.md` once). Everything below runs from `ui/apps/chat`.

Use it when a change in `ui/apps/chat` or `ui/lib/ui` is visible: a new or changed screen, flow or component. Skip it for refactors with no visible effect.

AGENTS.md still applies: the contributor writes the PR description and owns the script. `demo:pr` only fills the managed `## Demo` section.

## Pick the target

| Change in | Record | How |
| --------------------------------- | ---------------------- | --------------------------------------------- |
| Llama Chat flow or screen | the app, on vite dev | default |
| Llama UI component (`ui/lib/ui`) | its Storybook story | `test.use({ story })` + `DEMO_STORYBOOK=ui` |
| Llama Chat component in isolation | Chat's Storybook story | `test.use({ story })` + `DEMO_STORYBOOK=chat` |

A Storybook story is the harness page for a component: if none shows the state you need, add a story next to the others (`tests/stories/` in that package) rather than a new page.

## Setup (once)

```bash
cd ui/apps/chat && npm ci
```

```bash
npx playwright install chromium
```

Also `ffmpeg`, and `npm ci` in `ui/lib/ui` for Llama UI stories. `demo:pr` needs `gh` (edit rights on the PR) and, to link the video, an `hf` login plus `DEMO_STORAGE` and `DEMO_STORAGE_PUBLIC_URL` in `ui/apps/chat/.env` (see `.env.example`).

## 1. Demo

### Write the script

Scripts get `test` and `expect` from `helpers.ts`. Drive the page with `demo.*` (visible cursor, human timing), assert with `expect`, set the scene with `test.use`:

- `demo.click`, `demo.hover`, `demo.type`, `demo.press`, `demo.scroll`, `demo.pause`, `demo.caption(text)`, `demo.zoom(target)` (pinch onto a small detail).
- `test.use({ settings: {...} })` seeds UI settings, `startPath: '/#/...'` starts on a route.
- Fixtures, so no real data or third-party server is needed:
- **MCP servers:** two fixture servers (Weather, Notes) are always configured; see `demo/fixtures/demo-config.ts`.
- **Conversations:** `traces: ['long-chat', 'agentic-debugging']` imports built-in chats (`demo/fixtures/traces.ts`); other names come from `DEMO_TRACES_DATASET`.
- **Models and replies:** `mockServer: { models: ['org/Model-GGUF:Q4_K_M'], reply: 'Markdown **reply**' }` answers llama-server's endpoints in the browser (`demo/fixtures/llama-server.ts`): the model list, props, load, tools and a streamed reply. Nothing reaches a server and no model loads. Use it for any demo that sends a message.
- **Stories:** `story: 'ui-forms-switch--default'`, the id from the Storybook URL (`?path=/story/<id>`).

Without `mockServer` the app talks to the llama-server at `DEMO_SERVER_ORIGIN` (default `:9931`, started from `build/bin/llama-server` when none runs). Sending a message there loads a model: do not do that on someone's running server without asking.

### Try it locally

Save it as `demo/<name>.demo.ts` (git-ignored), importing from `./helpers.ts`, and run:

```bash
npm run demo:record -- demo/<name>
```

Add `DEMO_STORYBOOK=ui` (or `chat`) when the script has a `story`. `DEMO_SCALE=1` records faster while iterating. The mp4 lands in `demo/videos/`; check a few frames before going further:

```bash
ffmpeg -loglevel error -ss 3 -i demo/videos/<name>.mp4 -frames:v 1 demo/videos/<name>-3s.png
```

`npm run demo:codegen` opens Playwright codegen on `:5180` for a first draft of the selectors.

### Put it in the PR

Paste the script into the PR description as a `ts demo` block, inside the last collapsible block of Additional information:

````md
<details>
<summary>Demo script</summary>

```ts demo
test("demo", async ({ demo, page }) => {
// ...
});
```

</details>
````

Then record it at the PR head and write the `## Demo` section (between `<!-- demo:start -->` and `<!-- demo:end -->`):

```bash
npm run demo:pr -- <number>
```

- Each `test()` becomes its own video, shown in a table under its title. Group tests with their own `test.use` in `test.describe(() => { ... })`.
- In a PR block, the helpers import is replaced with `test` and `expect`, so reach fixtures through `test.use` options only.
- A PR on a fork: `DEMO_GITHUB_REPO=<owner>/llama.cpp`. A PR by someone else: read its diff and demo block, then pass `--trust <sha>`.

## 2. Test

Turn the same flow into a check in `tests/e2e/<feature>.e2e.ts`, which CI runs (`npm run test:e2e`) against the built app on `:8181`, with no llama-server:

- `@playwright/test` instead of the helpers: `locator.click()` / `fill()` for `demo.*`, keep every `expect`, drop captions and pauses.
- `mockLlamaServer(page, {...})` before `page.goto`, since nothing serves the API.
- `importTraces(page, file)` and `generateTrace(name)` seed conversations, as in a demo.

```bash
npx playwright test tests/e2e/<feature>.e2e.ts
```

Locally it uses the installed Chrome with its sandbox on; in CI, Playwright's Chromium.

For a component on its own, a `play` function on its story is the check instead (as in `ui/apps/chat/tests/stories/app/PwaRefreshAlert.stories.svelte`): both packages run their stories as vitest browser tests, `npm run test:ui` in `ui/apps/chat` and `npm test` in `ui/lib/ui`.

## 3. Document

- **Docs videos:** put the script in the storage target as `scripts/docs/<name>.demo.ts` (import `'../helpers.ts'`), then `npm run demo:docs` records every docs script and uploads `videos/docs/<name>.{mp4,gif}`. `DEMO_STORAGE=file:///some/dir` tries it without a bucket.
- **Screenshots:** in any demo, `await page.screenshot({ path: 'demo/videos/<name>.png' })` for the frame, or `await locator.screenshot({ path })` for one component, rendered at `DEMO_SCALE` (2x by default). Paths are relative to `ui/apps/chat`.
- **Component docs** belong in the package's Storybook: a story per state, prose in a `.svx` page (see `ui/apps/chat/AGENTS.md`).

## Worked example

A demo of chat search, then a reply from the mocked server, then a Llama UI story, in one PR block (record with `DEMO_STORYBOOK=ui npm run demo:pr -- <number>`):

```ts
test.describe(() => {
test.use({ traces: ["long-chat", "agentic-debugging"] });

test("Search finds a chat by its title", async ({ demo, page }) => {
await demo.click(page.getByLabel("Expand navigation"));
await demo.click(page.getByRole("button", { name: /^Search/ }));

const search = page.getByRole("dialog", { name: "Search" });

await demo.type(
search.getByRole("combobox", { name: "Search" }),
"flaky",
);

const result = search.getByRole("option", {
name: /Fix the flaky cache tests/,
});

await expect(result).toBeVisible();
await demo.click(result, 1200);
await expect(page.getByRole("main")).toContainText(
"All 48 cache tests pass now",
);
});
});

test.describe(() => {
test.use({
mockServer: {
reply: "Rust is a systems language focused on **safety** and *speed*.",
},
});

test("A reply from a mocked server, no model loaded", async ({
demo,
page,
}) => {
await demo.type(
page.getByRole("textbox", { name: "Type a message..." }),
"What is Rust?",
);
await demo.press("Enter", 1500);
await expect(page.getByRole("main")).toContainText("focused on safety");
});
});

test.describe(() => {
// a small viewport frames a small component, recorded at the same scale
test.use({
story: "ui-forms-switch--default",
viewport: { height: 450, width: 720 },
});

test("A Llama UI story: Switch", async ({ demo, page }) => {
const off = page.getByRole("switch").first();

await demo.click(off, 900);
await expect(off).toBeChecked();
});
});
```

The same search flow as a check is `tests/e2e/search-chats.e2e.ts`.

## Pitfalls

- **Ports:** vite dev `:5180` (`DEMO_UI_PORT`), fixture MCP `:8765`, llama-server `:9931`, Storybook `:6007` (`DEMO_STORYBOOK_PORT`), e2e `:8181`. Every server is reused when something already answers there, even another checkout's: stop stale servers, or pick other ports. `demo:pr` picks free ports itself.
- **Chrome sandbox:** demos record in Playwright's bundled Chromium with the sandbox on (headless installed Chrome ignores the viewport in screencasts); local e2e runs in the installed Chrome with the sandbox on. Do not turn the sandbox off to make a run pass: a PR demo runs that PR's code.
- **Waits:** wait on state, never on time. `expect(locator).toBeVisible()` / `toContainText()` before acting on something that appears late; `demo.pause` is pacing for the viewer only. The first load of a cold dev server reloads the page once; the harness waits for it before recording, so do not add sleeps for it.
- **Selectors:** use roles and accessible names. Some names match twice (`Expand navigation` is both the logo and the button: use `page.getByLabel('Expand navigation')`); scope to a dialog with `page.getByRole('dialog', { name })`. A strict-mode error names both matches.
- **Real server:** without `mockServer`, the model list and any reply come from the real llama-server and differ per machine. Mock them for anything a viewer should see the same way twice.
- **Preview size:** keep a demo short (under ~20 s per test): the GIF preview must stay under 5 MB, so a long one is shrunk until it is hard to read.
3 changes: 3 additions & 0 deletions ui/apps/chat/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ VITE_PUBLIC_SERVER_ORIGIN='http://localhost:8033'
# DEMO_SERVER_ORIGIN='http://localhost:9931'
# DEMO_LLAMA_SERVER='../../../build/bin/llama-server'
# DEMO_COLOR_SCHEME='light'
# Also start a Storybook for story demos: ui (Llama UI) or chat (Llama Chat)
# DEMO_STORYBOOK='ui'
# DEMO_STORYBOOK_PORT='6007'
# Device pixel ratio demos render and record at (1440x900 viewport)
# DEMO_SCALE='2'
# Longer side of the GIF preview, in pixels; it shrinks further to stay under 5 MB
Expand Down
1 change: 1 addition & 0 deletions ui/apps/chat/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
These add to the repository's [AGENTS.md](../../../AGENTS.md) for work in `ui/apps/chat`: **Llama Chat**, the chat client that llama-server bundles. **Llama UI**, the library, is `ui/lib/ui` (`@ggml-org/llama-ui`): Chat imports its tokens, fonts and base components from there (`@ggml-org/llama-ui/ui/<name>`) and keeps no copy of them. `DESIGN.md` and the Storybook live here; the brand book, `BRAND.md`, and the brand and communications pages are in [ggml-org/llama-brand](https://github.com/ggml-org/llama-brand), with their own Storybook. The agentic loop is `ui/lib/harness` (`@ggml-org/llama-harness`).

- **Design:** [DESIGN.md](DESIGN.md) holds the tokens, roles and rules, written for agents. Follow it for anything visual. Its token sections (`colors`, `typography`, `rounded`) are generated from the CSS with `npm run design:tokens`, and CI checks they are current; the rest is written by hand, and when it and the CSS disagree, the CSS wins and DESIGN.md is updated.
- **Demos, checks and docs media:** [skills/ui-demo](../../../skills/ui-demo/SKILL.md) records a PR's demo video, turns its flow into an e2e check and captures docs videos and screenshots, for Llama Chat and Llama UI alike.
- **Stories:** a story lives with the code it shows. Llama UI's (its foundations, base components, app components and Model UI) are in `ui/lib/ui/tests/stories`, titled `Llama UI/...`, and run as the library's tests; this app's Storybook shows them too. Llama Chat's are in `tests/stories/<area>`, one folder per `src/lib/components/app/<area>` (`chat/` has one per Chat folder), titled `Llama Chat/<Area>/<Component>`. An accessibility story sits next to its component's, as `<Component>.a11y.stories.svelte`, titled `.../<Component>/Accessibility`. The story helpers both Storybooks use (the Editorial the prose pages are set in, `StateGallery`) live with the library's stories and are imported here as `$llama-ui-stories/...`; reuse them rather than copying them here.
- **Releases:** `@ggml-org/llama-chat` and `@ggml-org/llama-ui` are versioned by hand and published by CI when a new version reaches `master` (see [Releases](../../README.md#releases)). Bump a version only when asked to release, never as a side effect of a change; Chat's version and its alias's (`alias/package.json`) always move together.
- **Components:** the `SKILL.md` next to a component folder (such as [src/lib/components/app/SKILL.md](src/lib/components/app/SKILL.md)) holds that folder's conventions.
Expand Down
16 changes: 15 additions & 1 deletion ui/apps/chat/demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,18 @@ Records every `scripts/docs/*.demo.ts` from the storage target and uploads `vide

Scripts get `test` and `expect` from `helpers.ts` without importing them. `demo.click`, `demo.hover` and `demo.type` glide a visible cursor to the target before acting, with timing varied like a person's (seeded per demo, so re-takes match); `demo.scroll` turns the mouse wheel over a target; `demo.zoom` pinches in on a control or a point to show a detail too small to read in the frame, holds, and pinches back out (with the fingertips drawn on a touch device); `demo.caption` shows a caption at the top of the frame. Use `test.use({ settings, startPath })` to seed other settings or start on another route.

## A mocked llama-server

`test.use({ mockServer: { models: ['org/Model-GGUF:Q4_K_M'], reply: 'A **Markdown** reply' } })` answers llama-server's endpoints in the browser (`fixtures/llama-server.ts`): the model list, props, model loading, tools and a streamed reply. Nothing reaches the server and no model loads, so a demo that sends a message looks the same on every machine. `tests/e2e` checks call `mockLlamaServer(page)` directly, since the e2e build runs without a server.

## Storybook stories

`test.use({ story: 'ui-forms-switch--default' })` records a Storybook story (the id from its `?path=/story/<id>` URL) instead of Llama Chat, from the Storybook that `DEMO_STORYBOOK` adds to the servers: `ui` for Llama UI (`ui/lib/ui`), `chat` for Llama Chat's. It runs on `DEMO_STORYBOOK_PORT` (default `:6007`). A small component reads better at a smaller `viewport` than zoomed in.

```bash
DEMO_STORYBOOK=ui npm run demo:record -- demo/<name>
```

## Conversations

Demos that need existing chats import them with `test.use({ traces: ['long-chat'] })`, through Settings > Import/Export before the video starts. Traces use the export format (llama.app JSONL), so the Hub shows them as Llama traces:
Expand All @@ -130,4 +142,6 @@ Demos that need existing chats import them with `test.use({ traces: ['long-chat'

To try a script locally, put it in `demo/` as `<name>.demo.ts`, importing from `./helpers.ts`, and run `npm run demo:record -- demo/<name>`. `npm run demo:codegen` opens Playwright codegen against `:5180` for a first draft.

The config starts, or reuses when already running, the fixture MCP servers on `:8765`, `llama-server` on `:9931` (router mode, `--tools all`) and the UI's vite dev server.
The config starts, or reuses when already running, the fixture MCP servers on `:8765`, `llama-server` on `:9931` (router mode, `--tools all`), the UI's vite dev server and, with `DEMO_STORYBOOK`, a Storybook.

For agents, [skills/ui-demo](../../../../skills/ui-demo/SKILL.md) walks through a demo, an e2e check and docs media for one change.
15 changes: 14 additions & 1 deletion ui/apps/chat/demo/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,12 @@ async function record(scripts: string[], uiDir = UI_DIR): Promise<Video[]> {
],
{
cwd: UI_DIR,
env: { ...process.env, DEMO_UI_DIR: uiDir, DEMO_UI_PORT: String(port) },
env: {
...process.env,
DEMO_STORYBOOK_PORT: String(await freePort()),
DEMO_UI_DIR: uiDir,
DEMO_UI_PORT: String(port)
},
stdio: 'inherit'
}
);
Expand Down Expand Up @@ -273,6 +278,14 @@ async function recordPr(number: string, trustedSha?: string): Promise<void> {
run('git', ['worktree', 'add', '--detach', worktree, sha]);
symlinkSync(join(UI_DIR, 'node_modules'), join(worktree, 'ui/apps/chat/node_modules'));

// Chat builds Llama UI from source, and DEMO_STORYBOOK=ui runs its Storybook
if (existsSync(join(UI_DIR, '../../lib/ui/node_modules'))) {
symlinkSync(
join(UI_DIR, '../../lib/ui/node_modules'),
join(worktree, 'ui/lib/ui/node_modules')
);
}

const name = `pr-${number}`;
const script = writeDemoScript(name, source);

Expand Down
63 changes: 63 additions & 0 deletions ui/apps/chat/demo/fixtures/llama-server.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/**
* A canned llama-server for demos and e2e checks that should not need a real one, or a model:
* routes its endpoints in the browser, so nothing reaches a server and no model loads.
*/

import type { BrowserContext, Page } from '@playwright/test';

export interface MockServerOptions {
/** Model ids the router lists, all loaded. */
models?: string[];
/** The assistant's reply to every message, streamed a word per chunk. */
reply?: string;
}

function completionStream(reply: string): string {
const words = reply.match(/\S+\s*/g) ?? [];
const chunk = (delta: object, finish: null | string = null) =>
`data: ${JSON.stringify({ choices: [{ delta, finish_reason: finish, index: 0 }], object: 'chat.completion.chunk' })}\n\n`;

return [
chunk({ role: 'assistant' }),
...words.map((word) => chunk({ content: word })),
chunk({}, 'stop'),
'data: [DONE]\n\n'
].join('');
}

/** Call before the page opens; a demo sets `test.use({ mockServer: {...} })` instead. */
export async function mockLlamaServer(
target: BrowserContext | Page,
{
models = ['ggml-org/gemma-3-4b-it-GGUF:Q4_K_M'],
reply = 'Hello! How can I help?'
}: MockServerOptions = {}
): Promise<void> {
const data = models.map((id) => ({
id,
object: 'model',
owned_by: 'llamacpp',
status: { value: 'loaded' }
}));

await target.route(/\/props(\?|$)/, (route) =>
route.fulfill({
json: { build_info: 'mock', model_path: 'none', role: 'router', ui_settings: {} }
})
);
await target.route(/\/(v1\/)?models(\?|$)/, (route) =>
route.fulfill({ json: { data, object: 'list' } })
);
await target.route(/\/models\/(load|unload)$/, (route) =>
route.fulfill({ json: { success: true } })
);
await target.route(/\/models\/sse$/, (route) => route.abort());
await target.route(/\/tools(\?|$)/, (route) => route.fulfill({ json: [] }));
await target.route(/\/v1\/streams\/lookup$/, (route) => route.fulfill({ status: 404 }));
await target.route(/\/v1\/chat\/completions$/, (route) =>
route.fulfill({
body: completionStream(reply),
headers: { 'content-type': 'text/event-stream' }
})
);
}
Loading
Loading