Repository navigation
feat(harbor): load a llama-swap model from the SWITCH tab - #286
Merged
Merged
Conversation
T-Crypt
marked this pull request as ready for review
November 2, 2026 04:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 6 of the switchboard plan: LOAD on the harbor's SWITCH tab. A paired device taps LOAD on a llama-swap model that is not loaded, and the model shows loading until llama-swap reports it ready.
How it works
POST /v1/berth/load {"engine","model"}. It takes only an engine withtype = "llama-swap"(any other is a 409) and a model that engine lists (any other is a 404). It answers 202 and, in the background, sendsGET {url}/upstream/{model}/health. llama-swap has no load endpoint, and any request under/upstream/MODEL/starts that model. A second load of the same model while one is under way joins the first. Each segment of the model ID is escaped on its own, so an ID such asorg/modelreaches llama-swap as two path segments.loading. It is set while a load from this gateway is under way, or when llama-swap reports the modelstarting. It is never set on a model that is already loaded. For an up llama-swap engine, the board reads llama-swap's/runningitself instead of berthd's copy, which is up to 5 s old. Each engine row gets itstype, which the plan's data shape already listed.POST /v1/board/loadbehind the same gates as switch and stop (cross-site 403, noX-Berth403, not paired 401). It requires a model, and a switch or stop never passes a model on.llama-swap v257 on aphotic has
upstream.ignorePaths, which arrived in v229. Its default only matches static files (.js,.json,.css, images,.txt), so/healthloads the model. The live run below confirms it.Proof
internal/gateway/load_test.goruns against a fake llama-swap whose/upstreamholds until the test releases it. It covers the four refusals (none reaches llama-swap), the 202 body, the escaped path/upstream/org/gemma%204/health,loadingon the board during the load and off after it, one upstream request for two loads, and each engine'stype. The board table test adds the cases forstarting, a load in flight, and a load in flight for a model that is already loaded. Each of nine mutations failed a test: the type check, the list check, the empty model, joining a second load, segment escaping, clearing after the load, the reader'sstarting, never loading a loaded model, and the in-flight flag.{"engine","model"}with the bearer key, and a switch drops a straymodel. Three mutations, all failed a test.?demo, 51 checks pass, including one LOAD on the cold model only, "loading" on the starting one, no LOAD on a non-llama-swap engine, and a 44 px LOAD button.go build,gofmt,go vet,go test ./..., the Windows build and vet, and golangci-lint (0 issues) pass locally.Live on aphotic
Qwen3.6-35B-A3B-IQ4-CoderFastafter an unload.GET /upstream/MODEL/healthanswered 200{"status":"ok"}after 2.26 s./runningshowedstartingat 0.1 s andreadywhen the answer arrived, so llama-swap holds the request until the model is ready. Warm, it answered in 2 ms, three times out of three. The 2.26 s is a load from the page cache, because the model had been loaded a minute before. A load from disk takes longer.POST /v1/berth/loadanswered 202. The first live run showed a gap: the board read loading at 0.01 s, cold again at 2.31 s, and loaded at 3.87 s. The gateway cleared its in-flight load when llama-swap answered, butloadedcame from berthd's reader, which reads/runningevery 5 s. The fix commit makes the board read llama-swap's/runningitself. After it, the board reads loading at 0.01 s and loaded at 2.31 s with no cold step, and the harbor's board shows the same. A test now covers this gap, and three mutations of the live read each fail it.strataengine gets 409, an unknown model 404, and the harbor withoutX-Berth403.