Skip to content

ui: run chat against more llama.cpp servers, added by URL or paired through a tunnel - #35

Draft
allozaur wants to merge 32 commits into
allozaur/feat/projectsfrom
allozaur/feat/servers
Draft

allozaur wants to merge 32 commits into
allozaur/feat/projectsfrom
allozaur/feat/servers

Conversation

@allozaur

@allozaur allozaur commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Overview

Chat can run against more than the server the app was loaded from: other llama.cpp servers are added by URL, or paired by an access code or its QR code and reached through a tunnel, and every server's models show up in the selector and the Models page. On a phone, Servers is a page of its own, like Settings and MCP. Pairing is experimental and off by default, behind a setting.

Additional information

Part 8 of 13 of a work-in-progress stack on the fork, based on allozaur/feat/projects. Its diff is only this layer, and it is written on the design system's tokens and components (#22, lower in the stack).

Since 13 November the stack has MCP as an extension (#61) below this layer. Two things here follow from that: Servers sits after the contributed MCP entry in the sidebar footer, so the order stays Models, MCP, Servers, Settings; and the page of an extension starts the extensions in its own load, because this layer moves the rest of the startup after the load.

The same work as ggml-org#29586, rebased onto this stack from its newer local branch (20 October: the rename to servers, the tunnel and pairing). The upstream PR and its branch are unchanged.

Pairing still sends the access code's pass code to whichever peer answers on the public tracker, so it sits behind "Enable remote server pairing" (Settings, General), marked Experimental and off by default, with a note naming that gap. While it is off, servers are added by URL only and no tunnel opens. A mutual challenge-response, where neither side sends the pass code, is the planned fix.

What changed in the rebase

The work was written before master squash-merged Discover (ggml-org#29584) and the Models Manager as a route (ggml-org#30988), and before this stack, so each commit was replayed with only its own change applied onto the current code.

  • The old Models Manager helpers the first commit carried (overrides, load defaults, draft helpers) were left out: master had already reworked or moved them. The model configuration layer above brings back the ones its pane needs.
  • hydrateStores (migrations, settings, servers) now runs before the stack's initStores and loadAppRoute.
  • The chat stream keeps the stack's finish-reason tracking: chat-wire.ts reports finish_reason.
  • With the manage-models dialog gone in master, "Manage Servers" is a button and a view on the Models page.
  • Servers sit in the sidebar footer between MCP and Settings.
  • Dropped from "show every provider's models in the manager": the draft filter, the "use as draft" row action and per-model overrides on load, which belong to the configuration pane.
Commits (31)
  • ui : add the providers data layer
  • ui : route chat through the active provider
  • ui : add the manage providers view
  • ui : show every provider's models in the manager
  • ui : resolve the layout probes through the providers api url
  • test : seed the picker drawer through the providers data layer
  • ui : keep the selector list open on provider backends
  • ui : show the selected model of the compat backends
  • ui : hide the chat template on the remote providers
  • ui : offer the compat providers a selection instead of a load
  • ui : keep the table status working while a provider is active
  • ui : keep llama.cpp servers as the only providers
  • ui : address remote providers through a tunnel
  • ui : pair a provider by code or camera
  • ui : cover the provider transport with tests
  • ui : rename the provider layer to servers
  • ui : rebuild the server card around what it reports
  • ui : recover a dropped channel when the tab comes back
  • ui : show the access code as a QR code
  • ui : give servers a dialog of their own
  • ui : add a server from a drawer on a phone
  • test : cover the servers surface
  • ui : keep a server's API key when it is read back
  • ui : refuse an endpoint that is not a llama.cpp server
  • ui : offer pairing where no server can answer
  • ui : read a remote router's status feed
  • ui : adapt the servers work to the stack
  • ui : open Servers as a page on a phone, a dialog elsewhere
  • ui: keep local-only requests on the local server, fix hidden models
  • ui : put remote server pairing behind an experimental setting
Testing
  • Type-check passes on this layer with 0 errors; 246 client and 795 unit tests pass (needs uqr@0.1.3, which the QR code commit adds).
  • Unit tests cover the pairing setting: off by default, marked Experimental with its note, no tunnel and no Remote server tab while it is off, and a tunnel once it is on. The pairing client tests switch it on.
  • Lint and formatting are clean on every touched file.
  • In the browser: the app loads, the sidebar footer lists Models, MCP, Servers and Settings, and the Servers dialog shows this server with its models and "Add new server".
  • In a phone viewport, Servers in the sidebar and the chat's no-server banner open #/servers, with the menu key and a title like the Settings and MCP pages, and the sidebar marks it; on desktop it stays a dialog with no change of route. DESIGN.md lists it with the other phone pages.
  • Not tried here: adding or pairing a remote server, and chatting through one.

Requirements

@allozaur
allozaur added this pull request to stack #37 November 2, 2026 14:32
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 2, 2026
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 62dbc6b to 8bda0fc Compare November 3, 2026 10:02
@allozaur
allozaur removed this pull request from stack #37 November 3, 2026 10:02
@allozaur
allozaur added this pull request to stack #41 November 3, 2026 10:03
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 8bda0fc to 4af2ddd Compare November 9, 2026 10:05
@allozaur
allozaur removed this pull request from stack #41 November 9, 2026 10:06
@allozaur
allozaur added this pull request to stack #42 November 9, 2026 10:06
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 4af2ddd to 39b0d62 Compare November 9, 2026 10:37
@allozaur
allozaur force-pushed the allozaur/feat/servers branch 3 times, most recently from 7fbabe8 to e7a2c6b Compare November 9, 2026 12:44
@allozaur
allozaur removed this pull request from stack #42 November 9, 2026 12:45
@allozaur
allozaur added this pull request to stack #44 November 9, 2026 12:45
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from e7a2c6b to 233d012 Compare November 9, 2026 14:35
@allozaur
allozaur removed this pull request from stack #44 November 9, 2026 14:36
@allozaur
allozaur added this pull request to stack #46 November 9, 2026 14:37
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 233d012 to 8be6ea1 Compare November 9, 2026 15:06
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 8be6ea1 to 6a2834b Compare November 9, 2026 15:24
@allozaur
allozaur force-pushed the allozaur/feat/servers branch from 6a2834b to c4d1026 Compare November 9, 2026 16:03
allozaur and others added 30 commits November 17, 2026 23:35
A provider is added from a preset card or by hand: the dialog probes the
endpoint, reads a refusal as a llama.cpp server that wants a key, and
shows the connection test as a status block. Presets carry their official
artwork and a saved backend keeps its branding through the favicon
fallback. The manager dialog gains its third view, and the settings save
stops clobbering keys that other surfaces write.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The table grows a section per enabled provider, each with its own mark,
error and loading state, and a provider filter with live repo counts.
Rows gain their capability gates back, the draft column with its
use-as-draft action, and the quant badge names the provider on its rows.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The key check reads the providers api url and waits for the server
capabilities, and the model status feed survives tab switches instead
of tearing down on every one. Split from the shell polish: the layout
depends on the providers layer.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The models store derives its list from the backends now, so the drawer
test seeds the active models instead of assigning the derived list, and
runs inside the tooltip provider the app shell supplies.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The active backend of an external provider has no /props, so its role
degrades to MODEL while the selector still has a list to show. The
trigger opened the manager instead of the list.

Gate the trigger on the selectable list instead of the router role.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
A compat backend serves the selection itself and reports no load state,
so the selected model sat inside its provider section, possibly cut off
by the display cap. Give it a leading section of its own, in the
selector list and the manager table, the way loaded models lead the
llama-compat ones.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
A remote provider serves its own template and reports no /props, so the
section would never show one.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The pane passed no canToggleLoad, so the load button defaulted to
visible for models of backends that cannot load or unload.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
The status cell branched on the active backend's role, so rows collapsed
to the served-by placeholder once a provider was the selected backend,
and the state lookup resolved the backend by raw-name first match, which
a same-named provider catalog entry shadowed: loading a loaded local
model then failed with 'model is already running'.

Branch the cell per row instead, resolve the state from the local feed
first and the backends' own listings after, and start the local feed
while a remote backend is active: the feed reports the local server's
models, so the local server's role decides.

Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash
Drop the OpenAI-compatible backend dialect from the providers work: every
backend now speaks the llama.cpp API, so there is nothing to negotiate.

Goes with it: the protocol enum and its capability and wire-quirk tables,
the adapter registry (it had a single implementation, and both protocols
resolved to it), the chat and models path overrides, the endpoint probe
that classified a URL as llama or OpenAI, and the hosted-provider presets
with their recommendation block. ChatService now builds llama.cpp requests
directly and decodes its stream through chat-wire, keeping the timings the
server reports instead of the wall-clock fallback a foreign endpoint needs.

The gates that read those capabilities collapse to their llama.cpp branch:
load, unload, slots, props, the resumable-stream endpoints and pre-encode
are simply available. A row of another server loads through that server,
and the switch that keeps local server state now keys on the built-in
backend rather than on a protocol.
A remote llama.cpp server is one the UI reaches over a WebRTC data channel
instead of a base URL. It stays a provider: it is listed, named, enabled and
disabled, and its models appear in the selector like everyone else's.

What changes is where a provider's requests are addressed. A remote provider
has no URL of its own, so it is given a synthetic origin whose host carries its
id (`https://<id>.remote.llamacpp.invalid`, the suffix being reserved so the
host can never resolve). Every service already builds its URLs through `apiUrl`,
so nothing in the services had to learn about tunnels: `getBackendBaseUrl` hands
back the synthetic origin, and a fetch wrapper installed in `hydrateStores`
routes a request by that host instead of by the page origin.

Routing on the address is what makes this safe to run next to other providers.
The first remote-access build tunneled everything aimed at the page origin, so
one tunnel was the most it could hold, and a remote provider with a URL of its
own would have slipped past it. Now a request aimed at a tunnel origin with no
live tunnel is refused out loud, and because the origin cannot resolve, a call
site that ever bypasses the resolver dies in the browser rather than reaching
the server that happens to serve this UI. A request queued while the handshake
ran is refused the same way if the channel died underneath it.

Tunnels open lazily: only the provider the app opens with is handshaked during
startup, the rest when one of their models is picked or Connect is hit on their
card. A provider that is disabled or removed loses its data channel, and its
code - which is a credential - is dropped with it.

Three things this forces into the open, all of them pinned to the built-in
provider back when no second provider could exist:

- `validateApiKey` probed the active provider with the key of the local one, so
  a remote provider answering 401 raised the API key splash over a credential
  belonging to another machine.
- `streamTool` ran tools on the local server while the model asking for one
  could be anywhere. A tool now runs where its tokens are generated.
- settings export scrubbed `apiKey` and MCP headers but exported the provider
  list whole, keys included. Remote providers now leave the export entirely:
  without their code they name nothing, and with it they are a key.

The tunnel itself comes from ggml-org#24577 (ngxson), with the base-path stripping
dropped: a request already carries the path the host's server serves.

Co-Authored-By: ngxson <ngxson@gmail.com>
The providers view gains the second kind of provider. A remote one shows its
code masked with a reveal, reports the tunnel and the server behind it as two
things that can fail - Connecting, N models, Disconnected, or Not connected
with a Connect button - and forgets its code when it is deleted.

Pairing is a code, and 40 characters of near-nonsense is a poor thing to type on
a phone, so a browser that ships a native BarcodeDetector gets a Scan button
that reads the QR code llama-server prints with --connect. No dependency, no
camera unless the user asks, and everywhere the detector is missing the field is
simply filled by hand.

Pairing saves the provider and then opens its tunnel, because the tunnel belongs
to the provider and has to be able to find it. A failed handshake therefore
leaves the provider in place with a retry on its card rather than rolling the
pair back: the code the user pasted is worth keeping.

With no local server and no provider the view says so plainly - this page has no
server of its own - and offers the code first, which is the state a build served
from the network opens in. That replaces the blocking splash of the first remote
build: the app boots, the built-in provider is dropped once its probe fails, and
what is left is an empty provider list that knows what to ask for.
The addressing rules are the cheap part to check and the expensive part to get
wrong: a provider id that stops round-tripping through its host, a code alphabet
that quietly accepts a pasted `I`, or a stored list that lets a URL provider
claim the tunnel host all mean a request goes somewhere it should not.

The client test drives the installed transport itself and asserts the part that
matters most - a refused request left nothing on the server serving the UI - and
the unit test pins the shapes the transport reads: codes, origins, and the
provider list that decides what may be routed at all.
The provider and backend vocabulary is retired across the data layer, the
stores, the components and the tests: the list, the view and the noun are
servers.

Assisted-by: DeepSeek-V4.1-Flash
A local card shows the host side of a server started with --connect, the
status words name the state instead of hiding it in a tooltip, a stalled
handshake can be dropped, and the access code can be copied off the card.

Assisted-by: DeepSeek-V4.1-Flash
A backgrounded tab on iOS or Android loses its peer connection, which leaves
the card on a failure the user can only clear by hand. Coming back now asks
for the channels that were already wanted, once the network has settled.

Assisted-by: DeepSeek-V4.1-Flash
The code a host prints can be scanned off the screen by the phone that pairs
with it, and the settings export says it carries access codes.

Assisted-by: DeepSeek-V4.1-Flash
The models dialog stays desktop-only, so a phone had no way into its server
list; the sidebar now carries a Servers entry that opens the list on its own.

Assisted-by: DeepSeek-V4.1-Flash
The add and pair form becomes a bottom sheet on a phone: its actions stay in
thumb reach and its fields scroll under them, while a desktop keeps the
dialog.

Assisted-by: DeepSeek-V4.1-Flash
The card's state table, the tunnel recovery and a handshake the user gives up
on, the QR round trip through the browser's own detector, and the phone's way
into the servers dialog.

Assisted-by: DeepSeek-V4.1-Flash
The parser dropped apiKey, so a key typed in the add or edit form was written
to the settings and never seen again: every request went out without it and a
key-protected server answered 401. A paired server keeps its key too, which
is what makes --connect together with --api-key possible.

Assisted-by: DeepSeek-V4.1-Flash
The UI drives llama.cpp servers only: it loads and unloads models, reads the
router feed and its chat template. /props is asked once the URL settles, and a
definite answer that it is something else blocks saving the server. An
endpoint that does not answer, or guards /props with a key, stays unknown, so
a working server behind a reverse proxy is never locked out.

Assisted-by: DeepSeek-V4.1-Flash
The chat banner is the surface left when nothing can serve a request, and it
stopped at a retry that cannot succeed. It now carries the door to the servers
dialog, where a server is paired.

Assisted-by: DeepSeek-V4.1-Flash
A load on a server behind a tunnel was a spinner with no progress, since the
feed was pinned to the local origin. The active server's own feed is read at
its origin while it is reachable, and an operation that waits on a load keeps
its polling as the fallback, so a feed that stops or never starts cannot hang
a row.

Assisted-by: DeepSeek-V4.1-Flash
The sidebar entry test renders the footer items through onAction, the way
the stack's SidebarNavigationActions takes them, and the files the replay
touched get their import order, class member order and formatting back.

Assisted-by: Claude (Opus 5.5 Medium)
On a phone the sidebar and the chat's no-server banner open #/servers, with the menu key and a
title like the Settings and MCP pages.

Assisted-by: Claude (Opus 5.5 Medium)
The CORS proxy, its auth header, streamed tool calls and the API key check
followed the active server, so with a remote server active, MCP traffic went
through someone else's machine and streamed tools ran where the tool list did
not come from. Pin them to the local server, like the tool list.

Hidden models: read ids saved before server-qualified ids as the local
server's, and check the active server's raw ids against qualified ones, so
hidden models stay hidden and out of the automatic pick. A conversation's
model now stays on the active server when that server has it too.

Assisted-by: Claude (Opus 5.5 Medium)
Pairing over WebRTC still sends the pass code to whichever peer answers on
the public tracker, so it ships off by default as an Experimental setting
that names that gap and the planned mutual challenge-response. While it is
off the servers form offers URLs only and no tunnel opens; turning it off
closes the ones already open.

Assisted-by: Claude (Opus 5.5 Medium)
Transcription, speech and the realtime socket go to the server in use,
through apiUrl, and the models that hear or speak come from its list.
A websocket is not tried for a remote server, which a browser socket
can not reach with its tunnel or key.

Assisted-by: Claude (Opus 5.5 Medium)
With no local llama-server, or with it switched off, the tools store
drops llama-server's tool list. The MCP servers it runs (stdio), which
the MCP extension lists from that same listing, now go with it.

Assisted-by: Claude (Opus 5.5 Medium)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation server/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant