Skip to content

feat: a size per card for a model split across cards - #291

Merged
T-Crypt merged 2 commits into
nextfrom
T-Crypt/needs-per-card
Nov 2, 2026
Merged

T-Crypt merged 2 commits into
nextfrom
T-Crypt/needs-per-card

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Refs #289

#289 asks how Berth handles a llama.cpp model split across several cards with --tensor-split. Most of it already worked. berth run --gpu gpu0,gpu1 holds one lease on both cards, grants them together, and starts the engine with CUDA_VISIBLE_DEVICES in the order named. The gap was sizing: --needs-mib was one size for every card, so with an uneven split the lighter card reserved as much as the heavier one. docs/design.md already describes "a lease that covers several cards with a size on each". This makes the code match it.

What changes for users

  • needs_mib = [18000, 6000] on an engine entry with gpu = "gpu0,gpu1", or berth run --needs-mib 18000,6000, reserves 18000 MiB on gpu0 and 6000 MiB on gpu1. The room left on gpu1 can serve another engine without a drain.
  • One value still sizes every card, and a config that has one writes back exactly as before.
  • A list that does not match the cards is refused, in the config check, in berth run, and in berthd.
  • docs/leasing.md has a new section, "Split one model across cards", with a llama.cpp --tensor-split example and how to size each card from berth status.

How it works

  • config.MiBs is the size list. It reads a number or a list in TOML, JSON and YAML, and 18000,6000 on the command line.
  • On the wire, needs_each carries the list and needs carries the largest size. A berthd that predates needs_each reads only needs, so it reserves too much on the smaller card, never too little.
  • The server checks the list against the cards it picked. AcquireAll and ReattachAll build each card's share from its own size, so a re-attach after a berthd restart keeps the sizes.
  • The gateway passes the list to the berth run it starts.

Proof

  • Build, gofmt, vet, go test ./... under America/Los_Angeles and UTC, the Windows build and vet, and golangci-lint (0 issues) all pass.
  • An end-to-end test runs berth run --gpu gpu1,gpu0 --needs-mib with two sizes against a real berthd with two cards. Each card's share holds its own size, in the order named. Two sizes for one card are refused with exit 75.
  • A broker test grants a 16 GiB ComfyUI lease beside an 18 GiB + 6 GiB split on the 6 GiB card, with no drain. It fails when the per-card size is dropped.
  • Ten mutations, one per new branch, were each caught by a test.
  • Not run on real hardware. aphotic has one NVIDIA card, so a real two-card split needs a tester.

A model split unevenly across cards (llama.cpp --tensor-split 3,1) needs
a different amount on each. needs_mib and berth run --needs-mib took one
size for every card, so the lighter card reserved as much as the heavier
one and nothing else could use the room left on it.

needs_mib now reads a number or a list (needs_mib = [18000, 6000]), and
--needs-mib reads "18000,6000", one size per card in the order gpu names
them. The run sends the list as needs_each and the largest size as needs,
so a berthd that predates needs_each reserves too much on the smaller
card, never too little. berthd refuses a list that does not match the
cards, and each card's share of the lease reserves its own size. One
value sizes every card, as before, and writes back as a plain number.
@T-Crypt
T-Crypt merged commit 4a504ae into next Nov 2, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/needs-per-card branch November 2, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant