diff --git a/FLOW.md b/FLOW.md index e50ffd7b8..629b8f8b8 100644 --- a/FLOW.md +++ b/FLOW.md @@ -771,6 +771,21 @@ Two shapes that surprise, both learned the hard way: (anomalyco/opencode#45424) which this fork's vendored core does not have yet: against an engine without it *and* without the config override above, those models read as runnable and still fail to dispatch. Drop the id list from `COMPATIBLE_PROVIDERS` if that PR is rejected. +- **A Claude subscription is not a provider, and no feature should pretend otherwise.** Anthropic + does not allow Pro/Max to drive another app, and OpenRouter's BYOK takes API keys only. Claude + reaches a card by API key, direct or through OpenRouter (whose BYOK is also how Bedrock/Vertex + credits pay for it). README "Using Claude" is the user-facing account. +- **A local model is declared, with four fields the engine gets wrong by default** (`lib/local.ts`): + a `local-` id (a models.dev id such as `lmstudio` is hidden until a key is stored), `tools: true` + (default off), a limit (default `{0,0}`), and a **zero price row** — priced at nothing, so the + unpriced-never-$0 rule and a gauntlet's cap both still hold. Never special-case local in + `usage.ts`; the config price is the whole mechanism. +- **Every run has a spend cap** (`spendCapOf`; absent means the $5 default, unlike the toggles, + because absent-means-off would leave the common case uncapped). Non-gauntlet runs hard-stop in + `enforceCap` on the repriced step and end `error` with `RunLog.capped`; a gauntlet keeps its + graceful `exhausted()` path at its own cap. The pre-run estimate in `graph/estimate.ts` is the + one guessed number here — it stays a range, and seat advice (`weak-seat`) fires only on an + inversion inside the canvas, never on a model merely being cheap. - `bun test` has **no `localStorage`** (it is a browser global). When persisting a preference the browser stores there — custom roles, the default-model preference (`graph/default-model.ts`), the walkthrough-dismissed flag — make a Solid signal the source of truth and treat diff --git a/packages/flow/README.md b/packages/flow/README.md index 48380c5ff..a2bf6c21f 100644 --- a/packages/flow/README.md +++ b/packages/flow/README.md @@ -321,6 +321,75 @@ served to the browser. | `GET /flow/api/zen-models` | model ids opencode zen really serves, or `null` if unreadable | | `GET /flow/api/repackage` | which OpenAI-compatible providers the global config already repackages | | `POST /flow/api/repackage` | repackage them there (fixed id list only, backed up, restart required) | +| `GET /flow/api/local` | local model servers, whether each answers, what it serves, what is declared | +| `POST /flow/api/local` | `{ id, url? }` — declare that server's models at $0 (backed up, restart required) | +| `DELETE /flow/api/local/:id` | remove a declared local server | + +### Using Claude + +Claude runs on a card through the paid API, billed per token. There are two +ways to reach it: + +1. **An Anthropic API key** from `console.anthropic.com`. Connect **Anthropic** + in the provider panel and paste it. The runner routes `@ai-sdk/anthropic` + directly, so nothing else is needed. +2. **Through OpenRouter.** Connect **OpenRouter**, then **Repackage** it. It + bills OpenRouter credits, or — with OpenRouter's BYOK (bring your own key) + setting — your own Anthropic key, or an **AWS Bedrock or Google Vertex** + key. That last one is how Claude spend comes out of cloud credits: this + engine cannot route Bedrock or Vertex on its own, but OpenRouter can. + +A Claude Pro or Max **subscription cannot be used**. Anthropic does not allow a +consumer subscription to drive another application, and OpenRouter's BYOK takes +API keys only, so routing through it does not change that. + +To keep a Claude run cheap: put the strong model where it pays off — the card +that dispatches, the critic, the synthesizer — and a cheaper one (Haiku, +Sonnet) on the cards that do the work. Preflight points out the reverse (a +`weak-seat` warning) and never changes a model itself. The spend panel shows how +much of the input came from the provider's prompt cache, which Anthropic bills +at a tenth of the input rate. + +### Local models + +Ollama, LM Studio, or any server that speaks OpenAI chat (llama.cpp's server, +vLLM) can run a card for nothing. Start the server and pull or load a model; the +provider panel's **Local models** group lists what each one serves, and a click +declares those models in the global config (`lib/local.ts`) and restarts the +engine. Clicking again after pulling a new model picks it up. + +What is written, and why each part is not optional: + +- Ids are prefixed `local-` (`local-ollama/qwen3`). The catalog hides a provider + with a models.dev id until a key is stored, and `lmstudio` is one. +- `tools: true`, because an undeclared model has tool calling switched off and + a card with no tools cannot read a file. +- A context limit, because the engine's default is zero. LM Studio reports the + context each model is loaded with (`/api/v0/models`) and that is used; other + servers report none, so 32k is written. Raise it on the model in the config + file; a later refresh keeps the edit. +- Only chat models. The OpenAI listing mixes in embedding models; LM Studio's + own route says which is which, and elsewhere an id naming `embed` is dropped. +- A **price of zero**, written as a real price row. The model is priced, so + spend reads `$0` and a gauntlet's cap still works — the rule that an unpriced + model never shows as `$0` is unchanged. + +A small local model is weakest in the seat that writes a control block every +turn (the orchestrator), which is where preflight's router warning fires for the +same reason. Use one where a card does the work. + +### Spend cap and estimate + +Every run has a dollar cap, **$5 unless the canvas says otherwise** (`cap:` in +the runbar; a gauntlet's own cap replaces it). The engine reprices on every step +and stops the run the moment priced spend reaches it; the log records `capped` +and the run ends as an error, not as `stopped`, because nobody decided it. A +model with no published price makes the cap partial, and preflight says so. + +Beside it, the runbar shows a rough range before anything is spent +(`graph/estimate.ts`): card turns counted from the graph, each priced between a +short turn and a heavy one at list price. The turn count is exact; the turn size +is a guess, which is why it is a range and the tooltip says what it assumed. `POST /api/session/{id}/wait` answers `503 Session wait is not available yet` on this server build, so idleness is derived from `/api/session/active` plus a diff --git a/packages/flow/lib/local.test.ts b/packages/flow/lib/local.test.ts new file mode 100644 index 000000000..200313e0c --- /dev/null +++ b/packages/flow/lib/local.test.ts @@ -0,0 +1,139 @@ +import { describe, expect, test } from "bun:test" +import { CUSTOM_LOCAL_ID, LOCAL_LIMIT, declare, declared, discover, normalizeUrl, runtimeFor, undeclare } from "./local" + +const ollama = runtimeFor("local-ollama")! + +describe("declare", () => { + test("writes a v2 provider with tools on, a limit, and a zero price row", () => { + const value = declare({ $schema: "x" }, ollama, ["llama3.2", "qwen3"]) + const entry = value.providers["local-ollama"] + expect(entry.api).toEqual({ type: "aisdk", package: "@ai-sdk/openai-compatible", url: "http://localhost:11434/v1" }) + expect(Object.keys(entry.models)).toEqual(["llama3.2", "qwen3"]) + expect(entry.models.qwen3.capabilities.tools).toBe(true) + expect(entry.models.qwen3.limit).toEqual(LOCAL_LIMIT) + expect(entry.models.qwen3.cost).toEqual([{ input: 0, output: 0, cache: { read: 0, write: 0 } }]) + expect(value.provider).toBeUndefined() + }) + + // One v1 key sends the whole file through the v1 migration, where + // `providers` is never read. + test("follows a v1 file's dialect", () => { + const value = declare({ plugin: ["./p.js"] }, ollama, ["llama3.2"]) + const entry = value.provider["local-ollama"] + expect(entry.npm).toBe("@ai-sdk/openai-compatible") + expect(entry.api).toBe("http://localhost:11434/v1") + expect(entry.models["llama3.2"].tool_call).toBe(true) + expect(entry.models["llama3.2"].cost).toEqual({ input: 0, output: 0, cache_read: 0, cache_write: 0 }) + expect(value.providers).toBeUndefined() + }) + + test("drops models the server no longer lists, keeps edits on the ones it does, and never keeps a price", () => { + const before = declare({}, ollama, ["old", "kept"]) + before.providers["local-ollama"].models.kept.limit = { context: 131_072, output: 8_192 } + before.providers["local-ollama"].models.kept.cost = [{ input: 9, output: 9, cache: { read: 0, write: 0 } }] + const after = declare(before, ollama, ["kept"]) + expect(Object.keys(after.providers["local-ollama"].models)).toEqual(["kept"]) + expect(after.providers["local-ollama"].models.kept.limit.context).toBe(131_072) + expect(after.providers["local-ollama"].models.kept.cost[0].input).toBe(0) + }) + + test("undeclare removes it from either dialect and leaves the rest", () => { + const v2 = undeclare(declare({ providers: { openai: {} } }, ollama, ["m"]), "local-ollama") + expect(v2.providers).toEqual({ openai: {} }) + const v1 = undeclare(declare({ plugin: [] }, ollama, ["m"]), "local-ollama") + expect(v1.provider).toEqual({}) + }) + + test("declared reads back what was written, and nothing that is not local", () => { + const value = declare({ providers: { openrouter: { api: { url: "https://x" } } } }, ollama, ["m"]) + expect(declared(value)).toEqual([{ id: "local-ollama", url: "http://localhost:11434/v1", models: ["m"] }]) + }) +}) + +describe("runtimeFor", () => { + test("accepts the fixed ids, with or without an address override", () => { + expect(runtimeFor("local-lmstudio")?.url).toBe("http://localhost:1234/v1") + expect(runtimeFor("local-ollama", "http://box:11434/v1/")?.url).toBe("http://box:11434/v1") + }) + + test("a custom server needs an http(s) address, and no other id is accepted", () => { + expect(runtimeFor(CUSTOM_LOCAL_ID)).toBeUndefined() + expect(runtimeFor(CUSTOM_LOCAL_ID, "file:///etc")).toBeUndefined() + expect(runtimeFor(CUSTOM_LOCAL_ID, "http://localhost:8000/v1")?.url).toBe("http://localhost:8000/v1") + expect(runtimeFor("openrouter", "http://localhost:8000/v1")).toBeUndefined() + }) + + test("normalizeUrl rejects what is not a url", () => { + expect(normalizeUrl("not a url")).toBeUndefined() + expect(normalizeUrl(42)).toBeUndefined() + }) +}) + +describe("discover", () => { + test("reads model ids off an OpenAI-compatible listing", async () => { + const server = Bun.serve({ + port: 0, + fetch: () => Response.json({ object: "list", data: [{ id: "qwen3" }, { id: "llama3.2" }, { id: "qwen3" }] }), + }) + const found = await discover(`http://localhost:${server.port}/v1`) + server.stop(true) + expect(found).toEqual({ running: true, models: ["llama3.2", "qwen3"] }) + }) + + // Measured against LM Studio: its OpenAI listing put an embedding model beside + // the chat ones, and its own /api/v0 route is what says which is which. + test("reads LM Studio's own listing to drop embedding models and take each loaded context", async () => { + const server = Bun.serve({ + port: 0, + fetch: (request) => + new URL(request.url).pathname === "/api/v0/models" + ? Response.json({ + data: [ + { id: "omnicoder-9b", type: "llm", loaded_context_length: 262_144 }, + { id: "ornith-1.0-9b", type: "llm", max_context_length: 131_072 }, + { id: "text-embedding-nomic-embed-text-v1.5", type: "embeddings" }, + ], + }) + : Response.json({ + data: [{ id: "omnicoder-9b" }, { id: "ornith-1.0-9b" }, { id: "text-embedding-nomic-embed-text-v1.5" }], + }), + }) + const found = await discover(`http://localhost:${server.port}/v1`) + server.stop(true) + expect(found.models).toEqual(["omnicoder-9b", "ornith-1.0-9b"]) + expect(found.context).toEqual({ "omnicoder-9b": 262_144, "ornith-1.0-9b": 131_072 }) + }) + + test("anywhere else, an id naming an embedding model is dropped", async () => { + const server = Bun.serve({ + port: 0, + fetch: (request) => + new URL(request.url).pathname === "/v1/models" + ? Response.json({ data: [{ id: "qwen3" }, { id: "nomic-embed-text:latest" }] }) + : new Response("no", { status: 404 }), + }) + const found = await discover(`http://localhost:${server.port}/v1`) + server.stop(true) + expect(found).toEqual({ running: true, models: ["qwen3"] }) + }) + + test("declare writes a model's measured context in place of the floor", () => { + const value = declare({}, ollama, ["big"], { big: 262_144 }) + expect(value.providers["local-ollama"].models.big.limit).toEqual({ context: 262_144, output: LOCAL_LIMIT.output }) + }) + + test("a server that is not there is reported as not running, not thrown", async () => { + const server = Bun.serve({ port: 0, fetch: () => new Response("") }) + const port = server.port + server.stop(true) + expect(await discover(`http://localhost:${port}/v1`)).toEqual({ running: false, models: [] }) + }) + + test("an error status is running, with the reason", async () => { + const server = Bun.serve({ port: 0, fetch: () => new Response("no", { status: 404 }) }) + const found = await discover(`http://localhost:${server.port}/v1`) + server.stop(true) + expect(found.running).toBe(true) + expect(found.error).toContain("404") + }) +}) diff --git a/packages/flow/lib/local.ts b/packages/flow/lib/local.ts new file mode 100644 index 000000000..a4774a9c5 --- /dev/null +++ b/packages/flow/lib/local.ts @@ -0,0 +1,236 @@ +import { COMPATIBLE_PACKAGE, isV1Config } from "./repackage" + +/** + * Makes a model running on the user's own machine — Ollama, LM Studio, or any + * server that speaks OpenAI chat — a provider every card can run on. + * + * Nothing in models.dev describes a local server, so the provider has to be + * declared in config, with its models listed by hand. That is what this writes, + * into the *global* config for the reason `lib/repackage.ts` gives: a session's + * drain reads the engine's own config, never the target project's. + * + * Four things a declared provider needs that are not obvious, each because the + * engine's default for it is wrong for a local model: + * + * - **An id models.dev does not know.** `catalog.ts` treats a provider as + * available only when it has a connection, *unless* no integration exists for + * its id at all. `lmstudio` is a models.dev id, so a provider named that is + * hidden until somebody stores a key a local server never asked for. Every id + * here is prefixed `local-` so none can collide, now or when models.dev grows. + * - **Tool calling switched on.** An empty model's capabilities say + * `tools: false`, and a card with no tools cannot read a file. + * - **A limit.** The empty default is `{ context: 0, output: 0 }`. The servers' + * `/models` listing does not report a context size, so a conservative one is + * written and the user can raise it in the file. + * - **A price of zero.** Written as a real price row, so `server/usage.ts` + * prices it exactly like any other model and a gauntlet's cap stays + * enforceable — the "unpriced is never $0" rule is untouched, because this + * model is priced, at nothing. + */ + +export type LocalRuntime = { id: string; name: string; url: string } + +/** Default addresses, from each runtime's own documentation. */ +export const LOCAL_RUNTIMES: LocalRuntime[] = [ + { id: "local-ollama", name: "Ollama", url: "http://localhost:11434/v1" }, + { id: "local-lmstudio", name: "LM Studio", url: "http://localhost:1234/v1" }, +] + +/** The one id a user-supplied address is written under. */ +export const CUSTOM_LOCAL_ID = "local-custom" + +export const LOCAL_PREFIX = "local-" + +/** Written for every model: the listing reports no context size, so this is a floor. */ +export const LOCAL_LIMIT = { context: 32_768, output: 8_192 } + +export function isLocalProvider(id: string) { + return id.startsWith(LOCAL_PREFIX) +} + +/** + * The base URL a user typed, normalised, or undefined when it is not one. + * + * Only http(s). Trailing slashes are dropped so `…/v1/` and `…/v1` are one + * address — the runner appends `/chat/completions` itself. + */ +export function normalizeUrl(input: unknown) { + if (typeof input !== "string") return undefined + const trimmed = input.trim().replace(/\/+$/, "") + const parsed = URL.canParse(trimmed) ? new URL(trimmed) : undefined + if (!parsed || (parsed.protocol !== "http:" && parsed.protocol !== "https:")) return undefined + return trimmed +} + +/** + * Model ids an OpenAI-compatible server lists at `GET /models`. + * + * Ollama, LM Studio, llama.cpp's server and vLLM all answer this route the same + * way. A server that is not running answers nothing, which is reported rather + * than thrown: "Ollama is not running" is the common case, not an error. + * + * That listing does not say what a model *is*, and servers list embedding + * models beside chat ones — measured against LM Studio, which listed + * `text-embedding-nomic-embed-text-v1.5` as a peer of its chat models. A card + * dispatched to one fails on its first turn. LM Studio answers a richer route, + * `/api/v0/models`, with each model's type and the context it was loaded with, + * so that is read when it is there; anywhere else an id naming an embedding + * model is dropped, which is the convention every runtime's catalog follows. + */ +export async function discover(url: string, fetcher: typeof fetch = fetch): Promise { + const response = await fetcher(`${url}/models`, { signal: AbortSignal.timeout(3_000) }).catch(() => undefined) + if (!response) return { running: false, models: [] as string[] } + if (!response.ok) return { running: true, models: [] as string[], error: `${url}/models answered ${response.status}` } + const body: any = await response.json().catch(() => undefined) + const listed: string[] = Array.isArray(body?.data) + ? body.data.map((entry: any) => entry?.id).filter((id: unknown): id is string => typeof id === "string" && !!id) + : [] + const detail = await lmStudioDetail(url, fetcher) + const models = [...new Set(listed)] + .filter((id) => (detail.has(id) ? detail.get(id)!.chat : !/embed/i.test(id))) + .sort() + const context = Object.fromEntries( + models.flatMap((id) => (detail.get(id)?.context ? [[id, detail.get(id)!.context!] as const] : [])), + ) + return { running: true, models, ...(Object.keys(context).length ? { context } : {}) } +} + +/** + * LM Studio's own listing, keyed by model id: whether it is a chat model, and + * the context it is loaded with (or can be, when it is not loaded yet). Empty + * for any other server, which answers this route with a 404. + */ +async function lmStudioDetail(url: string, fetcher: typeof fetch) { + const origin = url.replace(/\/v1$/, "") + const response = await fetcher(`${origin}/api/v0/models`, { signal: AbortSignal.timeout(3_000) }).catch(() => undefined) + const body: any = response?.ok ? await response.json().catch(() => undefined) : undefined + const entries: any[] = Array.isArray(body?.data) ? body.data : [] + return new Map( + entries + // An entry with no `type` is some other server echoing its OpenAI listing + // back on this route, and says nothing about what the model is. + .filter((entry) => typeof entry?.id === "string" && typeof entry?.type === "string") + .map((entry) => [ + entry.id as string, + { + chat: entry.type === "llm" || entry.type === "vlm", + context: Number(entry.loaded_context_length ?? entry.max_context_length) || undefined, + }, + ]), + ) +} + +/** Local providers this config declares, with the url and models each one lists. */ +export function declared(config: any) { + const v1 = Object.entries(config?.provider ?? {}).map(([id, entry]: [string, any]) => ({ + id, + url: entry?.api, + models: Object.keys(entry?.models ?? {}), + })) + const v2 = Object.entries(config?.providers ?? {}).map(([id, entry]: [string, any]) => ({ + id, + url: entry?.api?.url, + models: Object.keys(entry?.models ?? {}), + })) + return [...v1, ...v2].filter((entry) => isLocalProvider(entry.id) && typeof entry.url === "string") +} + +/** + * Declares one local provider with exactly these models, in whichever dialect + * the file is written in (see `isV1Config` — one v1 key sends the whole file + * through the v1 migration, where `providers` is never read). + * + * The models map is *replaced*, not merged: it is the server's own listing, and + * a model the user deleted from Ollama should stop being offered to cards. + * What the user edited on a model that is still listed — a raised context + * limit, most often — is kept. The price is not: it is always zero. + */ +export function declare(config: any, runtime: LocalRuntime, models: string[], context: Record = {}) { + const value = { ...config } + const limit = (id: string) => (context[id] ? { ...LOCAL_LIMIT, context: context[id] } : LOCAL_LIMIT) + if (isV1Config(config)) { + const previous = config?.provider?.[runtime.id]?.models ?? {} + value.provider = { + ...value.provider, + [runtime.id]: { + ...value.provider?.[runtime.id], + name: label(runtime), + npm: COMPATIBLE_PACKAGE, + api: runtime.url, + models: Object.fromEntries( + models.map((id) => [ + id, + { + name: id, + tool_call: true, + modalities: { input: ["text"], output: ["text"] }, + limit: limit(id), + ...previous[id], + cost: { input: 0, output: 0, cache_read: 0, cache_write: 0 }, + }, + ]), + ), + }, + } + return value + } + + const previous = config?.providers?.[runtime.id]?.models ?? {} + value.providers = { + ...value.providers, + [runtime.id]: { + ...value.providers?.[runtime.id], + name: label(runtime), + api: { type: "aisdk", package: COMPATIBLE_PACKAGE, url: runtime.url }, + models: Object.fromEntries( + models.map((id) => [ + id, + { + name: id, + capabilities: { tools: true, input: ["text"], output: ["text"] }, + limit: limit(id), + ...previous[id], + cost: [{ input: 0, output: 0, cache: { read: 0, write: 0 } }], + }, + ]), + ), + }, + } + return value +} + +/** Removes a local provider from either dialect. */ +export function undeclare(config: any, id: string) { + const value = { ...config } + if (value.provider?.[id]) { + value.provider = { ...value.provider } + delete value.provider[id] + } + if (value.providers?.[id]) { + value.providers = { ...value.providers } + delete value.providers[id] + } + return value +} + +/** The runtime an id names, with the address the caller supplied for a custom one. */ +export function runtimeFor(id: string, url?: string): LocalRuntime | undefined { + const known = LOCAL_RUNTIMES.find((runtime) => runtime.id === id) + const address = normalizeUrl(url) + if (known) return { ...known, url: address ?? known.url } + if (id === CUSTOM_LOCAL_ID && address) return { id, name: "Local server", url: address } + return undefined +} + +/** What the model menu heads this provider with. */ +function label(runtime: LocalRuntime) { + return runtime.id === CUSTOM_LOCAL_ID ? runtime.name : `${runtime.name} (local)` +} + +export type Discovered = { + running: boolean + models: string[] + /** Context each model is loaded with, when the server says (LM Studio does). */ + context?: Record + error?: string +} diff --git a/packages/flow/lib/store.test.ts b/packages/flow/lib/store.test.ts index a18164ada..5b2eb1696 100644 --- a/packages/flow/lib/store.test.ts +++ b/packages/flow/lib/store.test.ts @@ -1196,6 +1196,59 @@ describe("custom roles", () => { }) }) +/** Declaring a local model server also writes the global config — same isolation. */ +describe("routes: local model servers", () => { + let target: string + let server: ReturnType + + beforeEach(() => { + process.env.XDG_CONFIG_HOME = path.join(dir, "xdg") + target = path.join(dir, "xdg", "opencode", "opencode.json") + server = Bun.serve({ port: 0, fetch: () => Response.json({ data: [{ id: "qwen3" }] }) }) + }) + + afterEach(() => { + delete process.env.XDG_CONFIG_HOME + server.stop(true) + }) + + test("declares what the server serves, at $0, and asks for a restart", async () => { + const url = `http://localhost:${server.port}/v1` + const response = await call("POST", "/flow/api/local", { body: { id: "local-custom", url } }) + expect(response!.body).toMatchObject({ changed: true, models: ["qwen3"], restart: true }) + const entry = JSON.parse(await read(target)).providers["local-custom"] + expect(entry.api.url).toBe(url) + expect(entry.models.qwen3.cost[0]).toMatchObject({ input: 0, output: 0 }) + const status = (await call("GET", "/flow/api/local"))!.body as any + expect(status.runtimes.find((runtime: any) => runtime.id === "local-custom")).toMatchObject({ + running: true, + models: ["qwen3"], + declared: ["qwen3"], + }) + }) + + test("a server that is not answering is refused, and nothing is written", async () => { + const port = server.port + server.stop(true) + const response = await call("POST", "/flow/api/local", { body: { id: "local-custom", url: `http://localhost:${port}/v1` } }) + expect(response!.status).toBe(409) + expect((response!.body as any).error).toContain("nothing is answering") + await expect(read(target)).rejects.toThrow() + }) + + test("an id outside the fixed list is refused, so no provider or package can be named", async () => { + const response = await call("POST", "/flow/api/local", { body: { id: "openrouter", url: "http://localhost:1/v1" } }) + expect(response!.status).toBe(400) + }) + + test("removing one takes it out and leaves the rest of the file", async () => { + await call("POST", "/flow/api/local", { body: { id: "local-custom", url: `http://localhost:${server.port}/v1` } }) + const response = await call("DELETE", "/flow/api/local/local-custom") + expect(response!.body).toMatchObject({ changed: true }) + expect(JSON.parse(await read(target)).providers).toEqual({}) + }) +}) + /** * The one route that writes *outside* the project: repackaging a provider edits * opencode's global config, because that is the only file a run reads (see diff --git a/packages/flow/lib/store.ts b/packages/flow/lib/store.ts index a7ab05f88..3f8efea8f 100644 --- a/packages/flow/lib/store.ts +++ b/packages/flow/lib/store.ts @@ -6,6 +6,7 @@ import { recallPipeline, rememberPipeline, rememberProject } from "./last-sessio import { hasNativePicker, pickFolderNative } from "./native-picker" import type { Supervisor } from "./opencode-process" import { COMPATIBLE_PROFILES, globalConfigCandidates, repackage, repackaged } from "./repackage" +import { CUSTOM_LOCAL_ID, LOCAL_RUNTIMES, declare, declared, discover, isLocalProvider, runtimeFor, undeclare } from "./local" import { install, installed, uninstall } from "./dispatch-tool" import { checkpoint, dropCheckpoints } from "./checkpoint" import { dropScratch, openScratch } from "./scratch" @@ -63,6 +64,10 @@ import { zenModels } from "./zen" * already repackages * POST /flow/api/repackage -> { providers: [...] } repackages them there, so the * runner can dispatch their models + * GET /flow/api/local -> { path, runtimes: [{ id, name, url, running, models, + * declared }] } — local model servers and what each lists + * POST /flow/api/local -> { id, url? } declares that server's models as a $0 + * provider in the global config; DELETE /local/:id removes it * GET /flow/api/server -> { managed, running, url, command, pid, ... } — the * `opencode serve` this host proxies * POST /flow/api/server/restart -> restarts it, when this host owns it (loopback only) @@ -342,6 +347,25 @@ export async function handleFlow(paths: FlowPaths, request: FlowRequest): Promis } } + if (segments[0] === "local") { + if (method === "GET" && segments.length === 1) return ok(await localStatus()) + if (method === "POST" && segments.length === 1) { + const body = await request.json().catch(() => ({})) + // Only the fixed ids are accepted, and the package is always the + // OpenAI-compatible one, so no request can name an npm package to load. + const runtime = runtimeFor(String(body?.id ?? ""), body?.url) + if (!runtime) return { status: 400, body: { error: "name a local runtime, or give local-custom an http(s) url" } } + const result = await applyLocal(runtime) + if ("error" in result) return { status: 409, body: result } + return ok(result) + } + if (method === "DELETE" && segments.length === 2 && isLocalProvider(segments[1])) { + const result = await removeLocal(segments[1]) + if ("error" in result) return { status: 409, body: result } + return ok(result) + } + } + if (segments[0] === "server") { if (!request.serve) return { status: 501, body: { error: "this host does not track `opencode serve`" } } if (segments.length === 1 && method === "GET") return ok(await request.serve.status()) @@ -1126,6 +1150,64 @@ async function applyRepackage(ids: string[]) { }) } +/** + * Every known local runtime, probed, plus any declared at an address of the + * user's own. A runtime that is declared but not running is still listed — its + * cards will fail to dispatch, and the panel has to be able to say why. + */ +async function localStatus() { + const config = await readGlobalConfig() + const present = "error" in config ? [] : declared(config.value) + const custom = present.find((entry) => entry.id === CUSTOM_LOCAL_ID) + const runtimes = [ + ...LOCAL_RUNTIMES.map((runtime) => ({ ...runtime, url: present.find((entry) => entry.id === runtime.id)?.url ?? runtime.url })), + ...(custom ? [{ id: CUSTOM_LOCAL_ID, name: "Local server", url: custom.url as string }] : []), + ] + const probed = await Promise.all( + runtimes.map(async (runtime) => ({ + ...runtime, + ...(await discover(runtime.url)), + declared: present.find((entry) => entry.id === runtime.id)?.models, + })), + ) + return { path: config.target, runtimes: probed, ...("error" in config ? { error: config.error } : {}) } +} + +/** + * Lists what the server serves right now and declares exactly that. + * + * A server that is not running, or that serves nothing, is refused rather than + * written as an empty provider — a provider with no models is one the panel + * would show as connected and no card could use. + */ +async function applyLocal(runtime: { id: string; name: string; url: string }) { + const found = await discover(runtime.url) + if (!found.running) return { error: `nothing is answering at ${runtime.url} — start ${runtime.name} first` } + if (!found.models.length) + return { error: found.error ?? `${runtime.name} at ${runtime.url} lists no models — pull or load one first` } + return writeGlobal((value) => declare(value, runtime, found.models, found.context), { models: found.models }) +} + +async function removeLocal(id: string) { + return writeGlobal((value) => undeclare(value, id), {}) +} + +/** Rewrites the global config through `edit`, backing it up first. */ +async function writeGlobal(edit: (value: any) => any, extra: Record) { + const config = await readGlobalConfig() + if ("error" in config) return { error: config.error, path: config.target } + const value = edit(config.value) + if (JSON.stringify(value) === JSON.stringify(config.value)) + return { path: config.target, changed: false, restart: false, ...extra } + if (config.comments) return { error: CONFIG_HAS_COMMENTS, path: config.target } + return serialize(config.target, async () => { + const backup = config.raw === undefined ? undefined : await backupConfig(config.target, config.raw) + await fs.mkdir(path.dirname(config.target), { recursive: true }) + await writeAtomic(config.target, JSON.stringify(value, null, 2) + "\n") + return { path: config.target, changed: true, backup, restart: true, ...extra } + }) +} + const CONFIG_HAS_COMMENTS = "existing opencode.json has comments, and saving would strip them — remove the comments, or make this change in opencode.json by hand" diff --git a/packages/flow/src/app.tsx b/packages/flow/src/app.tsx index 940602df5..1ac312b7c 100644 --- a/packages/flow/src/app.tsx +++ b/packages/flow/src/app.tsx @@ -1,9 +1,11 @@ import { For, Show, createMemo, createSignal, onCleanup, onMount } from "solid-js" import { Canvas } from "./canvas/canvas" import { + DEFAULT_MAX_SPEND, depthOf, dispatchesOf, gauntletOf, + spendCapOf, MAX_DEPTH, MAX_DISPATCHES, MAX_ROUNDS, @@ -14,6 +16,7 @@ import { type Pipeline, type RunLog, } from "./graph/types" +import { estimate, estimateLabel } from "./graph/estimate" import { MCP_REACHES_SESSIONS } from "./graph/dispatch" import { layer, preflight, type Preflight } from "./graph/validate" import { isPipeline } from "./graph/pipeline-io" @@ -125,6 +128,11 @@ const ISOLATE_OPTIONS: SelectOption[] = [ { value: "off", label: "off", hint: "one shared working directory" }, { value: "on", label: "on", hint: "a git worktree per card, merged after each batch" }, ] +const CAP_OPTIONS: SelectOption[] = [1, 2, 5, 10, 25, 50, 100, 250, 500].map((amount) => ({ + value: String(amount), + label: `$${amount}`, + hint: amount === DEFAULT_MAX_SPEND ? "default" : undefined, +})) const VERIFY_OPTIONS: SelectOption[] = [ { value: "off", label: "off", hint: "the run ends when its cards do" }, { value: "on", label: "on", hint: "a reviewer card judges the result and the run reports its verdict" }, @@ -244,6 +252,14 @@ export function App() { * gating on `unlockedRows` here would block a first-timer's run outright. */ const unlockedModels = () => new Set(availableModels(providers()).map((model) => model.value)) + /** Base list price per model the catalog quotes one for — the estimate and the seat check read it. */ + const prices = createMemo( + () => + new Map( + providers().flatMap((row) => row.models.flatMap((model) => (model.price ? [[model.value, model.price] as const] : []))), + ), + ) + const runEstimate = createMemo(() => estimate(state.pipeline, prices())) /** * Every error this file reports goes through here rather than through @@ -792,6 +808,7 @@ export function App() { const pre = preflight(state.pipeline, { unlockedModels: unlockedModels(), engineReachable: state.engineReachable, + prices: prices(), }) setIssues(pre.blocking.length || pre.warnings.length ? pre : undefined) if (pre.blocking.length) { @@ -1294,6 +1311,37 @@ export function App() { onChange={(value) => actions.setIsolate(value === "on")} /> + {/* Every mode. A gauntlet shows its own cap in its settings button, + and that one wins (`spendCapOf`), so it is not offered twice. */} + + setCustomUrl(event.currentTarget.value)} + /> + + + +
Popular
{(row) => } @@ -291,9 +367,74 @@ export function ProvidersPanel(props: { ) } + /** + * One local server. A click sets it up — or, when it is set up, picks up any + * model pulled since — and the tag says which of those a click will do. + */ + function LocalRow(item: { runtime: LocalRuntimeStatus }) { + const fresh = () => { + const declared = item.runtime.declared + return !!declared && declared.length === item.runtime.models.length && declared.every((id) => item.runtime.models.includes(id)) + } + return ( +
+ + + + +
+ ) + } + function KeyForm(item: { row: ProviderRow }) { return (
+ +

+ Claude runs here on an API key from console.anthropic.com, billed per token. A + Claude Pro or Max subscription cannot be used by other apps. To pay from AWS or Google Cloud credits + instead, connect OpenRouter and add your Bedrock or Vertex key under its BYOK settings. +

+
+ +

+ Claude through OpenRouter bills OpenRouter credits, or your own Anthropic, Bedrock or Vertex key if you add + one under its BYOK settings. Repackage it after connecting so this engine can dispatch it. +

+

{props.spend?.steps ? costLabel(props.spend) : "—"} {formatTokens(props.spend ? tokenCount(props.spend) : 0)} tokens · {props.spend?.steps ?? 0} steps + + {(share) => ( + + {" "} + · {Math.round(share() * 100)}% cached + + )} + )