Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions docs/console-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,14 +307,34 @@ Not there yet:
- **Search on the services list.** swissd pages its list (at most 100 a page) and
takes no query; filtering one page on the client would miss the others. It
needs `?q=` on swissd's `/api/deployments`.
- **swiss's panels and the route directive texts are English.** They are swiss's,
and swiss's pages have no Chinese yet.
- **swiss's panels are English.** They are swiss's, and swiss's pages have no Chinese yet. Route option labels are translated by inferences; untranslated help and new options use the engine's text.
- **Cluster resources needs a recent swissd.** An older one (0.5.6, for one) has
no `/objects`: the tab says so, and the install track reports the route and
scaler as unreadable rather than absent.
- **Upgrade picks chart versions through `/api/catalog/:name/chart-versions`**,
which swissd has from 0.6.2 (swiss #13).

### Deploy route options

The deploy and upgrade form reads the running openresty's catalog instead of treating bundled defaults as deployment defaults (console #52; [agreed design](development/route-options-catalog.md)).

```text
DeploySheet -> GET /api/llm/route-options[?route=<existing>]
-> configured gateway -> /<route>/_route_opts
-> common grouped rows + advanced picker + runtime state
```

| Concern | Decision |
|---|---|
| Authorization | Console login and `get backends` for `llm`, the same grant as the inference proxy |
| New deployment | Probe existing ConfigMap routes for a catalog; never show their per-route values as the new deployment's |
| Upgrade | Query the recorded route, display `value` and `set` separately from saved overrides |
| Defaults | Live `fallback` supplies placeholders; no copied values are submitted automatically |
| Legacy or empty gateway | 404 uses bundled metadata with all fallback values stripped |
| Other failures | Surface a catalog warning; do not claim a deployment default |
| Text | Per-key locale translations where present; new options retain engine text |
| Ownership | Inferences adapts the Swiss metadata without editing the copied upstream module |

### Workload reads (`/api/k8s/...`)

Events and logs are Kubernetes data, not swiss's, so console serves them itself
Expand Down
61 changes: 61 additions & 0 deletions docs/development/route-options-catalog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Route Options Catalog

Design agreed from console issue #52 on 2026-10-12.

```text
DeploySheet -> GET /api/llm/route-options[?route=<existing>]
-> configured gateway -> /<route>/_route_opts
-> common grouped rows + advanced picker + runtime state
```

| Concern | Decision |
|---|---|
| API | Each configured backend exposes GET `<prefix>/route-options`; gateway backends select resolved ConfigMap routes, while static URLs already name a route |
| Authorization | Console login and `get backends` on the configured backend name |
| Request | Optional existing route; reject invalid names and unknown routes; without one try available routes until a catalog answers |
| Credentials | Do not forward the browser session, cookies or gateway API key to the public engine endpoint |
| Compatibility | 404 for an unsupported engine, missing backend, empty gateway or unavailable route; bundled metadata has no fallback values |
| Failures | Permission, transport and malformed response failures are surfaced, not silently treated as legacy engines |
| Form state | Fetching metadata never changes stored overrides; `value` and `set` describe the current runtime separately |
| Runtime identity | Only show runtime values for the canonical requested upgrade route, including the release-name fallback for older plans |
| Named controls | Preserve chart and SLO semantics, use live placeholders, and avoid duplicate rows |
| Advanced | Engine internals remain behind the add-setting picker, which respects available viewport height |
| i18n | Translate known keys; unknown keys use engine text |
| Swiss | Adapt bundled metadata in inferences without editing the copied Swiss module |

No CRD, chart value or configuration schema changes. Validation remains in the engine.

## Regression Coverage

| Scenario | Signal |
|---|---|
| SPA with no gateway backend | Catalog API returns 404 JSON instead of 200 HTML |
| Host defaults differ from bundled values | Render the live defaults, never copied placeholders |
| Existing route differs from a new deployment | Do not leak the probed route's runtime state into a new form |
| Upgrade without explicit route/serviceId overrides | Use the same canonical route for fetching and display |
| Unknown engine option | Render engine metadata without a console release |
| Optional boolean override | Preserve unset, true and false distinctly |
| Empty advanced setting | Omit the setting rather than sending its displayed default |
| Catalog response | Bound response size, reject malformed data and do not follow redirects |

## Real Engine Verification

Verification used a released `4pdosc/llm-openresty:0.1.26` image with an isolated catalog-only route and the existing preview application services. The route is not a working inference pool or a business deployment.

| Check | Result |
|---|---|
| Engine catalog | 16 common directives and 24 advanced directives |
| Different host defaults | TPS fallback 73 and floor fraction 0.61 appear as placeholders |
| Separate route state | TPS value 81 with set=true is distinct from fallback 73 |
| Proxy | Preserves the engine response; known routes return 200 and unknown routes 404 |
| Browser compose | Returns 200; live defaults and empty advanced settings remain omitted |
| Native mouse interaction | Advanced option is pointer-reachable; selection keeps the form open and the value unset |
| Local checks | Go suite, frontend suite, typecheck and production build pass |

The standalone browser check requires an auth-disabled preview, a real configured engine and the GLM sample model:

```sh
CONSOLE_URL=http://127.0.0.1:8080 node hack/check-route-catalog-live.mjs
```

It checks live placeholders, viewport containment, pointer hit testing and native selection. Acceptance is desktop-only. No install/apply operation was performed, and existing stored release history was not rewritten.
74 changes: 74 additions & 0 deletions hack/check-route-catalog-live.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";

const base = process.env.CONSOLE_URL;
assert.ok(base, "Set CONSOLE_URL to an auth-disabled preview with a real engine catalog");
const session = "route-catalog-live-check";
function browser(...args) {
const result = JSON.parse(execFileSync("agent-browser", ["--session", session, "--json", ...args], { encoding: "utf8", timeout: 40_000 }));
assert.equal(result.success, true, result.error);
return result.data;
}
const evaluate = (script) => browser("eval", script).result;

const response = await fetch(`${base}/api/llm/route-options`);
assert.equal(response.status, 200, "A real engine catalog must be configured");
const catalog = await response.json();
const options = new Map([...catalog.directives, ...catalog.advanced].map((d) => [d.key, d]));

try {
browser("open", base);
browser("set", "viewport", process.env.VIEWPORT_WIDTH ?? "1280", process.env.VIEWPORT_HEIGHT ?? "720");
evaluate('localStorage.setItem("rise-locale", "en-US")');
browser("open", `${base}/inferences/catalog/glm5.1?catalog=default`);
browser("wait", "--text", "Deploy");
const defaults = evaluate(`(async () => {
const frame = () => new Promise(requestAnimationFrame);
[...document.querySelectorAll("button")].filter((x) => x.textContent.trim() === "Deploy").at(-1).click();
for (let i = 0; i < 300 && !document.querySelector("#deploy-sec-routing-section"); i++) await frame();
const section = document.querySelector("#deploy-sec-routing-section");
if (!section) throw new Error("Deploy routing section did not render");
const sheet = section.closest('[data-slot="sheet-content"]');
const sheetRect = sheet.getBoundingClientRect();
if (sheetRect.left < 0 || sheetRect.right > innerWidth) throw new Error("Deploy sheet exceeds viewport width");
section.querySelector("button").click();
await frame();
const tpsLabel = [...document.querySelectorAll("label")].find((x) => x.textContent.includes("TPS fallback"));
const tps = document.getElementById(tpsLabel.htmlFor);
const floor = document.querySelector('input[aria-label="Adaptive floor fraction"]');
const trigger = section.querySelector('button[aria-haspopup="menu"]');
trigger.scrollIntoView({ block: "center" });
for (let i = 0; i < 12; i++) await frame();
trigger.click();
for (let i = 0; i < 12; i++) await frame();
const item = [...document.querySelectorAll('[role="menuitem"]')].find((x) => x.textContent.includes("rt_limit_factor"));
const menu = item.closest('[role="menu"]');
const menuRect = menu.getBoundingClientRect();
if (menuRect.top < 0 || menuRect.bottom > innerHeight) throw new Error("Advanced menu exceeds available viewport height");
const initial = item.getBoundingClientRect();
menu.scrollTop += initial.top - menuRect.top - (menu.clientHeight - initial.height) / 2;
for (let i = 0; i < 12; i++) await frame();
const rect = item.getBoundingClientRect();
const x = rect.left + rect.width / 2;
const y = rect.top + rect.height / 2;
if (document.elementFromPoint(x, y)?.closest('[role="menuitem"]') !== item) throw new Error("Advanced item is not pointer reachable");
return { tps: tps.placeholder, floor: floor.placeholder, tpsValue: tps.value, floorValue: floor.value, x, y };
})()`);
assert.equal(defaults.tps, options.get("tps_limit_tps").fallback);
assert.equal(defaults.floor, options.get("adaptive_cc_min_frac").fallback);
assert.equal(defaults.tpsValue, "");
assert.equal(defaults.floorValue, "");
browser("mouse", "move", String(Math.round(defaults.x)), String(Math.round(defaults.y)));
browser("mouse", "down");
browser("mouse", "up");
const selected = evaluate(`(() => {
const input = document.querySelector('input[id$="-rt_limit_factor"]');
if (!input) throw new Error("Native pointer selection did not add the option");
return { value: input.value, placeholder: input.placeholder };
})()`);
assert.equal(selected.value, "");
assert.equal(selected.placeholder, options.get("rt_limit_factor").fallback);
console.log(JSON.stringify({ realCatalog: true, route: catalog.route, hostDefaults: { tps: defaults.tps, floor: defaults.floor }, viewportAndNativePointer: true, unsetAdvancedPreserved: true }));
} finally {
browser("close");
}
33 changes: 21 additions & 12 deletions internal/gateway/gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ type Entry struct {
KeySource string
}

var ErrNoRoutes = fmt.Errorf("no session_route_<route>.conf entry")

// refresh is how long a resolved entrypoint is reused. Long enough that a busy
// Playground does not read the same two ConfigMaps per request, short enough that
// a deploy shows up without touching the console.
Expand Down Expand Up @@ -89,7 +91,7 @@ func (r *Resolver) Resolve(ctx context.Context) (*Entry, error) {
return nil, r.err
}

entry, err := r.resolve(ctx)
entry, err := r.resolve(ctx, false)
r.at = time.Now()
switch {
case err != nil && r.entry != nil:
Expand All @@ -111,7 +113,12 @@ func (r *Resolver) Resolve(ctx context.Context) (*Entry, error) {
return entry, nil
}

func (r *Resolver) resolve(ctx context.Context) (*Entry, error) {
// ResolveCatalog discovers existing routes without inference pinning or credentials.
func (r *Resolver) ResolveCatalog(ctx context.Context) (*Entry, error) {
return r.resolve(ctx, true)
}

func (r *Resolver) resolve(ctx context.Context, catalog bool) (*Entry, error) {
gw := r.backend.Gateway
if gw == nil {
return nil, fmt.Errorf("backend %s is not a gateway", r.backend.Name)
Expand Down Expand Up @@ -141,7 +148,7 @@ func (r *Resolver) resolve(ctx context.Context) (*Entry, error) {
port = config.DefaultGatewayPort
}

routes, source, err := r.routes(ctx, profile, profileRef)
routes, source, err := r.routes(ctx, profile, profileRef, catalog)
if err != nil {
return nil, err
}
Expand All @@ -155,30 +162,32 @@ func (r *Resolver) resolve(ctx context.Context) (*Entry, error) {
Header: profile.Route.Auth.HeaderName(),
Source: source,
}
if err := r.credential(ctx, profile, svcNS, entry); err != nil {
return nil, err
if !catalog {
if err := r.credential(ctx, profile, svcNS, entry); err != nil {
return nil, err
}
}
return entry, nil
}

// routes lists the routes to talk to: the explicit one, else every route in the
// route ConfigMap, aggregate ones (those serving several models) first. autoconfig
// writes one plain route per model, so a gateway with no aggregate route is the
// normal case, not an error.
func (r *Resolver) routes(ctx context.Context, p profile, profileRef string) (routes []string, source string, err error) {
// routes lists inference routes, or all configured routes for catalog discovery.
func (r *Resolver) routes(ctx context.Context, p profile, profileRef string, catalog bool) (routes []string, source string, err error) {
gw := r.backend.Gateway
ref := gw.ConfigMap
if ref == "" {
ref = p.Route.NginxConfigMap
}
if gw.Route != "" {
if gw.Route != "" && (!catalog || ref == "") {
source = ref
if source == "" {
source = profileRef
}
return []string{gw.Route}, source, nil
}
if ref == "" {
if catalog {
return nil, "", ErrNoRoutes
}
return nil, "", fmt.Errorf("no gateway.route and no route configmap to choose one from: set gateway.route")
}
ns, name, err := splitRef(ref, "")
Expand All @@ -203,7 +212,7 @@ func (r *Resolver) routes(ctx context.Context, p profile, profileRef string) (ro
}
}
if len(aggregates)+len(plain) == 0 {
return nil, "", fmt.Errorf("route configmap %s has no session_route_<route>.conf entry; is this the entrypoint's shared configmap?", ref)
return nil, "", fmt.Errorf("route configmap %s has %w; is this the entrypoint's shared configmap?", ref, ErrNoRoutes)
}
sort.Strings(aggregates)
sort.Strings(plain)
Expand Down
71 changes: 71 additions & 0 deletions internal/gateway/gateway_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package gateway

import (
"context"
"errors"
"log/slog"
"slices"
"strings"
Expand Down Expand Up @@ -99,6 +100,76 @@ func resolveOne(t *testing.T, cfg *config.Gateway, c *fakeCluster) (*Entry, erro
return NewResolver(b, c, slog.New(slog.DiscardHandler)).Resolve(t.Context())
}

func TestResolveCatalogIndependentOfInferencePin(t *testing.T) {
for _, pin := range []string{"video", "stale"} {
t.Run(pin, func(t *testing.T) {
c := fixture()
b := config.Backend{Name: "llm", Gateway: &config.Gateway{Profile: "llm/site-profile", Route: pin}}
r := NewResolver(b, c, slog.New(slog.DiscardHandler))
inference, err := r.Resolve(t.Context())
if err != nil {
t.Fatal(err)
}
catalog, err := r.ResolveCatalog(t.Context())
if err != nil {
t.Fatal(err)
}
if catalog.Base != "http://openresty.llm.svc:8080" || !slices.Equal(catalog.Routes, []string{"llm-gateway", "glm-5.1", "kimi-k2.6"}) || catalog.Key != "" {
t.Fatalf("catalog = %+v", catalog)
}
again, err := r.Resolve(t.Context())
if err != nil || again != inference || !slices.Equal(again.Routes, []string{pin}) || again.Key != "Bearer k1" {
t.Fatalf("inference pin or credential changed: %+v, %v", again, err)
}
})
}
}

func TestResolveCatalogWithoutCredentialSecret(t *testing.T) {
c := fixture()
c.secrets = nil
t.Setenv("CONSOLE_TEST_CATALOG_KEY", "secret")
for _, keyEnv := range []string{"", "CONSOLE_TEST_CATALOG_KEY"} {
b := config.Backend{Name: "llm", APIKeyEnv: keyEnv, Gateway: &config.Gateway{Profile: "llm/site-profile"}}
entry, err := NewResolver(b, c, slog.New(slog.DiscardHandler)).ResolveCatalog(t.Context())
if err != nil || entry.Key != "" || entry.KeySource != "" {
t.Fatalf("catalog required or exposed a credential: %+v, %v", entry, err)
}
}
}

func TestResolveCatalogRouteSources(t *testing.T) {
for _, tc := range []struct {
name string
gw config.Gateway
routes []string
err error
}{
{"explicit without configmap", config.Gateway{Service: "llm/openresty", Route: "model", Port: 9000}, []string{"model"}, nil},
{"no route source", config.Gateway{Service: "llm/openresty"}, nil, ErrNoRoutes},
{"empty configmap despite pin", config.Gateway{Service: "llm/openresty", ConfigMap: "llm/empty-conf", Route: "stale"}, nil, ErrNoRoutes},
{"missing configmap despite pin", config.Gateway{Service: "llm/openresty", ConfigMap: "llm/missing", Route: "stale"}, nil, errNotFound},
{"explicit configmap overrides profile", config.Gateway{Profile: "llm/site-profile", ConfigMap: "llm/many-routes-conf", Route: "stale"}, []string{"a", "b"}, nil},
} {
t.Run(tc.name, func(t *testing.T) {
b := config.Backend{Name: "llm", Gateway: &tc.gw}
entry, err := NewResolver(b, fixture(), slog.New(slog.DiscardHandler)).ResolveCatalog(t.Context())
if tc.err != nil {
if !errors.Is(err, tc.err) {
t.Fatalf("err = %v, want %v", err, tc.err)
}
return
}
if err != nil || !slices.Equal(entry.Routes, tc.routes) {
t.Fatalf("entry = %+v, err = %v", entry, err)
}
if tc.gw.Port == 9000 && entry.Base != "http://openresty.llm.svc:9000" {
t.Fatalf("explicit port ignored: %+v", entry)
}
})
}
}

func TestResolveFromProfile(t *testing.T) {
entry, err := resolveOne(t, &config.Gateway{Profile: "llm/site-profile"}, fixture())
if err != nil {
Expand Down
Loading
Loading