Skip to content

feat(harbor): load a llama-swap model from the SWITCH tab - #286

Merged
T-Crypt merged 5 commits into
nextfrom
T-Crypt/harbor-load-model
Nov 2, 2026
Merged

T-Crypt merged 5 commits into
nextfrom
T-Crypt/harbor-load-model

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Step 6 of the switchboard plan: LOAD on the harbor's SWITCH tab. A paired device taps LOAD on a llama-swap model that is not loaded, and the model shows loading until llama-swap reports it ready.

How it works

  • Gateway. A new POST /v1/berth/load {"engine","model"}. It takes only an engine with type = "llama-swap" (any other is a 409) and a model that engine lists (any other is a 404). It answers 202 and, in the background, sends GET {url}/upstream/{model}/health. llama-swap has no load endpoint, and any request under /upstream/MODEL/ starts that model. A second load of the same model while one is under way joins the first. Each segment of the model ID is escaped on its own, so an ID such as org/model reaches llama-swap as two path segments.
  • Board. Each model gets loading. It is set while a load from this gateway is under way, or when llama-swap reports the model starting. It is never set on a model that is already loaded. For an up llama-swap engine, the board reads llama-swap's /running itself instead of berthd's copy, which is up to 5 s old. Each engine row gets its type, which the plan's data shape already listed.
  • Harbor. POST /v1/board/load behind the same gates as switch and stop (cross-site 403, no X-Berth 403, not paired 401). It requires a model, and a switch or stop never passes a model on.
  • UI. On a paired device, a llama-swap row's model list has LOAD on each cold model, a half-filled glyph and "loading" on a model that is starting, and "loaded" on a loaded one.

llama-swap v257 on aphotic has upstream.ignorePaths, which arrived in v229. Its default only matches static files (.js, .json, .css, images, .txt), so /health loads the model. The live run below confirms it.

Proof

  • internal/gateway/load_test.go runs against a fake llama-swap whose /upstream holds until the test releases it. It covers the four refusals (none reaches llama-swap), the 202 body, the escaped path /upstream/org/gemma%204/health, loading on the board during the load and off after it, one upstream request for two loads, and each engine's type. The board table test adds the cases for starting, a load in flight, and a load in flight for a model that is already loaded. Each of nine mutations failed a test: the type check, the list check, the empty model, joining a second load, segment escaping, clearing after the load, the reader's starting, never loading a loaded model, and the in-flight flag.
  • Harbor tests: a load without a model is a 400, an unpaired load is a 401, a paired load forwards {"engine","model"} with the bearer key, and a switch drops a stray model. Three mutations, all failed a test.
  • In Chromium on the phone and the desktop, with this branch's harbor and ?demo, 51 checks pass, including one LOAD on the cold model only, "loading" on the starting one, no LOAD on a non-llama-swap engine, and a 44 px LOAD button.
  • go build, gofmt, go vet, go test ./..., the Windows build and vet, and golangci-lint (0 issues) pass locally.

Live on aphotic

  • llama-swap directly, with Qwen3.6-35B-A3B-IQ4-CoderFast after an unload. GET /upstream/MODEL/health answered 200 {"status":"ok"} after 2.26 s. /running showed starting at 0.1 s and ready when the answer arrived, so llama-swap holds the request until the model is ready. Warm, it answered in 2 ms, three times out of three. The 2.26 s is a load from the page cache, because the model had been loaded a minute before. A load from disk takes longer.
  • Through the gateway, with this branch installed. POST /v1/berth/load answered 202. The first live run showed a gap: the board read loading at 0.01 s, cold again at 2.31 s, and loaded at 3.87 s. The gateway cleared its in-flight load when llama-swap answered, but loaded came from berthd's reader, which reads /running every 5 s. The fix commit makes the board read llama-swap's /running itself. After it, the board reads loading at 0.01 s and loaded at 2.31 s with no cold step, and the harbor's board shows the same. A test now covers this gap, and three mutations of the live read each fail it.
  • Refusals on the live gateway: a strata engine gets 409, an unknown model 404, and the harbor without X-Berth 403.

@T-Crypt
T-Crypt marked this pull request as ready for review November 2, 2026 04:12
@T-Crypt
T-Crypt merged commit a5b3bc6 into next Nov 2, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/harbor-load-model branch November 2, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant