inference

This repository runs model servers on a 2x RTX A4000 box. A router puts them behind one address. Every model has a replica on each GPU, so the router can rotate a model's requests across both cards. The default stack is the MVP — reranker and voyage-embed; every optional service is a profile in its own compose file (see "Profiles").

The rules for measurement and tuning are in TUNING.md; the measured numbers are in docs/evaluation.md; the vocabulary is in docs/glossary.md.

The stack

flowchart TB
  client["Client<br/>HTTP, one address"]

  router["router · 8100<br/>Node.js, no dependencies<br/>backends from docker.sock<br/>re-read every 30s<br/><br/>POST one of:<br/>/v1/embeddings<br/>/rerank<br/>/v1/rerank<br/>/v1/decisions<br/>/v1/systemone<br/>/v1/predict<br/>/api/evaluate"]

  subgraph optional_profiles["optional profiles — one make up-* each"]
    direction LR
    qwen["qwen-embed · TEI<br/>8003 / 8004<br/>Qwen3-Embedding-0.6B"]
    bakeoff["bake-off rerankers · TEI<br/>bge · ms-marco<br/>gte-multilingual · granite<br/>8096-8099<br/>GPU 0 only, no replica"]
    dec["decision services · Python<br/>8041-8048<br/>omnijev /v1/systemone<br/>julia /v1/predict<br/>laya /v1/decisions<br/>agentjev /api/evaluate"]
    jina["jina · jina-embed<br/>Python · 8092-8095<br/>non-commercial<br/>eval only, never serve"]
  end

  subgraph default_stack["default stack — make up"]
    direction LR
    reranker["reranker · TEI<br/>8021 / 8022<br/>Alibaba-NLP/gte-reranker-modernbert-base"]
    voyage["voyage-embed · vLLM<br/>8001 / 8002<br/>voyageai/voyage-4-nano<br/>bf16: TEI gives NaN<br/>at fp16, OOM at fp32"]
  end

  ollama["ollama · 11434 · GGUF<br/>beside the router<br/>no discovery label<br/>open to the network"]

  router --> qwen
  router --> bakeoff
  router --> dec
  router --> jina
  router --> reranker
  router --> voyage
  client --> router
  client -.->|"direct, not routed"| ollama

  classDef entry fill:#e6eefa,stroke:#3a5a8a
  classDef core fill:#e7f4e9,stroke:#3a7d44
  classDef extra fill:#f6f1e4,stroke:#8a7a3a
  class client,router entry
  class reranker,voyage core
  class qwen,bakeoff,dec,jina,ollama extra

Loading

Every backend port binds to 127.0.0.1, so 8100 is the only way in from off-box. A port pair is one replica per GPU and the router alternates between them.

An advertised path routes only to the backends that advertised it, so a Python service answers its own surface while TEI and vLLM, which omit paths, take the router's static set.

The two servers are:

• reranker — the local reranker.
• voyage-embed — the embedder for the voyage-4-nano model.

Servers

Name	Port	GPU	Runtime	Model
reranker	8021	0	TEI	Alibaba-NLP/gte-reranker-modernbert-base
voyage-embed	8001	0	vLLM	voyageai/voyage-4-nano
router	8100	—	Node.js	—

voyage-embed uses vLLM, not TEI. voyage-4-nano is a bf16 model:

• At fp16, TEI returns NaN vectors for some inputs.
• At fp32, TEI runs out of memory.
• vLLM runs the model at bf16, and the vectors are correct.

Profiles

Five services start by default: router, plus a pair for each of the two models — reranker and voyage-embed — with the second replica on the other GPU. That is the MVP. Everything optional is a profile in its own compose file, and every profile has a pair of targets:

Profile	File	Services	Ports	GPU
rerankers	compose.rerankers.yml	bge-reranker, ms-marco, gte-multilingual, granite-reranker	8099, 8098, 8097, 8096	0
qwen-embed	compose.qwen-embed.yml	qwen-embed, qwen-embed-b	8003, 8004	0, 1
laya	laya/compose.laya.yml	laya (own Python runtime)	8043	0
agentjev	agentjev/compose.agentjev.yml	agentjev (own Python runtime)	8045	0
jina	jina/compose.jina.yml	jina (own Python runtime, non-commercial)	8095	0
jina-embed	jina-embed/compose.jina-embed.yml	jina-embed (own Python runtime, non-commercial)	8093	0
omnijev	omnijev/compose.omnijev.yml	omnijev (own Python runtime, Apache-2.0)	8041	0
julia	julia/compose.julia.yml	julia (own Python runtime, Apache-2.0)	8047	0
ollama	ollama/compose.ollama.yml	ollama (own runtime, beside the router, no label)	11434	1

Target	Effect
make up-rerankers / make down-rerankers	the four bake-off rerankers
make up-qwen-embed / make down-qwen-embed	the Qwen3 embedder pair (the index build model)
make up-laya / make down-laya	the Laya decision service
make up-agentjev / make down-agentjev	the AgentJev decision service
make up-jina / make down-jina	the Jina reranker (non-commercial, local dev)
make up-jina-embed / make down-jina-embed	the Jina embedder (non-commercial, local dev)
make up-omnijev / make down-omnijev	the OmniJev decision service (Apache-2.0)
make up-julia / make down-julia	the Julia-1 decision service (Apache-2.0)
make up-ollama / make down-ollama	the Ollama GGUF runner (reference only)
make up-all / make down-all	every profile, everything

Port bands, so each kind of server lives in its own mental space: 8001–8004 embeds, 8021–8022 rerankers, 8041–8048 decision services, 8080–8099 optionals and ad-hoc slots (the non-commercial jina pair lives here too: 8095/8094 and 8093/8092; pinned top-down from 8099, ad-hoc bottom-up from 8080), 8100 router.

Each up-* starts the default stack as well, so one command always leaves a router in front of what it started. Each down-* removes only its own profile's services — the router and the default models keep running. make up-all does not fit this box: every profile defaults to GPU 0, against voyage-embed's half-card reservation — bring profiles up selectively instead. GPU= moves a profile's services to another card (make up-laya GPU=1); empty means each profile's default. The compose command underneath, for the rerankers profile:

docker compose -f compose.yml -f compose.rerankers.yml \
  --profile rerankers up -d --build

Every Python profile takes a second replica on the other card, with an adjacent port pair: omnijev 8041/8042, laya 8043/8044, agentjev 8045/8046, julia 8047/8048, jina-embed 8093/8092, jina 8095/8094. make up-omnijev CONCURRENCY=2 enables the -b service (default CONCURRENCY=1 keeps one instance), and the router alternates requests between each pair — the same -b shape the default stack uses.

Every profiled service carries the tei.backend=1 label, so the router finds it with no config change — make models is the live union. Use the rerankers profile to compare rerankers with one another and with the paid APIs.

Quick start

make help (or bare make) lists every target and the make flags.

# 1. Create the model cache.
mkdir -p "$HOME/.hf-cache"

# 2. Build the router image and start the default stack (no `.env` needed,
# every variable has a default in the compose files).
make up

# 3. Show the model of each server.
make status

# 4. Show the model list of the router.
make models

# 5. Show the token rate and the queue of the vLLM engines while they work.
make throughput

Two ports are open to the network. The router on 8100 is the only way in for the serving fleet: clients use the router and never a server port, because every backend binds to localhost, so off-box traffic cannot bypass the router. Ollama on 11434 is the second. It sits beside the router rather than behind it and is published on every interface, so a client reaches it directly with the full /api/* surface — which has no authentication and covers pulling and deleting models, not just inference. OLLAMA_BIND=127.0.0.1 closes it to this box.

Router

router/ is a proxy with no dependencies. It does three tasks:

• It reports the model list of all servers.
• It sends each model-carrying POST to the server with the requested model.
• It normalizes /rerank's Cohere documents to TEI's texts.

The router reads the model of each server. It reads the TEI /info data, or the OpenAI /v1/models list of a vLLM server. It repeats the read every 30 seconds. So the router finds a new or changed server with no restart.

GET /health reports uptime and per-model and per-backend counts — requests, errors, p50, and which replica took each one. /healthz is an alias for the same body. Counts are kept for the models the router serves; a request naming an unknown model is answered 404 and logged, but it does not add an entry, so the map cannot be grown by probing.

Backends advertise the POST paths they serve in /info (paths), and the router unions those with its static set — a new surface routes with no router change. An advertised path routes only to the backends that advertised it, so a rolling upgrade never sends it to an un-upgraded replica. A backend that omits paths (TEI, vLLM) routes the static set only; malformed entries are ignored, and unlisted paths 404:

Service	Advertised paths
laya	/rerank, /v1/decisions
agentjev	/rerank, /api/evaluate
jina	/rerank
jina-embed	/v1/embeddings
omnijev	/v1/systemone
julia	/v1/predict

curl -s localhost:8100/v1/models
curl -s localhost:8100/v1/embeddings -H 'content-type: application/json' \
  -d '{"model":"voyageai/voyage-4-nano","input":["A brewer may sell beer."]}'

Change a model

The defaults live in the compose files — MODEL_VOYAGE_EMBED for voyage-embed (vLLM) and MODEL_RERANKER for the reranker (TEI). To change one, set it in .env (optional overrides only) and recreate. Each server loads one model at start, so a change is a recreate:

echo 'MODEL_RERANKER=BAAI/bge-reranker-v2-m3' >>.env
docker compose up -d voyage-embed reranker

For a short test, do not change the compose file. Start a free slot instead. The script finds a free GPU and a free port. The router finds the new slot.

make run MODEL=BAAI/bge-reranker-base
make status
make stop NAME=tei-baai-bge-reranker-base

Services

Each optional service documents itself beside its compose file:

• Laya — the decision model (laya profile, GPU 0, port 8043). Typed questions with calibrated probabilities; the candidate for the one rerank row that is currently a paid API.
• AgentJev — the decision model on Qwen3-0.6B (agentjev profile, GPU 0, port 8045). Boolean, choice, and score answers with full distributions and zero decoded tokens.
• Jina — the listwise reranker plus the jina-embed embedder sibling (jina and jina-embed profiles, ports 8095 and 8093). Non-commercial weights: local dev and eval only.
• OmniJev — the omni-modal decision model (omnijev profile, GPU 0, port 8041). Typed questions about one image with calibrated probabilities and explicit abstention; Apache-2.0, the one decision service that may serve.
• Julia — the decision model (julia profile, GPU 0, port 8047). The named-question interface on a 144.3M mmBERT-small encoder — text-only, CPU-capable, Apache-2.0.
• Ollama — the GGUF runner beside the router (port 11434, no discovery). Reference only.

Measured results

The codex retrieval evaluation numbers live in docs/evaluation.md: embedding models at 1024 vs 2048 dims, and a reranker bake-off. That table's local ordering is superseded for the serving choice — see docs/reranker-selection.md, which records the paired comparison and the decision. The fast tier is make eval-golden: the same bake-off over the 105-case golden subset (eval/), for regression.

Operations

Box setup and troubleshooting live in docs/operations.md: first start, NVIDIA prerequisites, image tags, the shared model cache, diagnostics, and gotchas.

Notes

Background reading, not runbooks: tuning (how to measure an indexing build), LoRA (cheap adaptation notes), vLLM (engine notes).