From 9cff3e7f647b3790baa8865741d6ebd1e164b10c Mon Sep 17 00:00:00 2001 From: Sijie Guo Date: Sat, 3 Oct 2026 00:32:58 -0700 Subject: [PATCH] fix: recover the local stack after a port clash Docker Engine 29.2 leaves a container whose port could not be bound without its network. Every later start of that container has loopback only, even once the port is free. Only a new container recovers. For the Local course that meant: - Lab 0 step 2: after `Bind for 0.0.0.0:8080 failed`, running local/engine.sh again (what --check says to do) failed with `registry-1 exited (1)`, and stopping the other program did not help. - Lab 0 step 1: after stopping the other program, `up` exited 0 and the step's check listed six services, but the container had no published port. local/engine.sh now removes the engine's containers that are not running before `ork local start`. The engine's data is in volumes. A rerun gives the same bind error while the port is taken, and a working engine once it is free. Troubleshooting gives a recovery that works for both stacks, covers a doctor that cannot reach a container that lost its network, and has a row for a missing python/.venv on the CLI path. Lab 0 steps 1 and 2 point to it. --- labs/local/00-set-up.md | 6 +++ labs/local/troubleshooting.md | 7 ++-- local/engine.sh | 12 +++++- local/lib.sh | 6 +++ local/tests/run.sh | 69 +++++++++++++++++++++++++++++++++-- 5 files changed, 93 insertions(+), 7 deletions(-) diff --git a/labs/local/00-set-up.md b/labs/local/00-set-up.md index de3fce5..6a84b32 100644 --- a/labs/local/00-set-up.md +++ b/labs/local/00-set-up.md @@ -41,6 +41,9 @@ The first run downloads about 4.4 GB of images. It starts six services - `risingwave`: streaming SQL. - `risingwave-mcp`: RisingWave's MCP server, the agent's SQL tools. +If it fails with a port already in use, see +[Troubleshooting](troubleshooting.md) before you run it again. + ### Check The six services are running. Before the step this prints nothing. @@ -84,6 +87,9 @@ header: Run `local/engine.sh` again whenever the engine has been restarted. It is safe to run at any time. +If it stops with `port is already allocated`, another program has a port the +engine needs, usually 8080: see [Troubleshooting](troubleshooting.md). + ### Check The engine is up, has a provider key, and can reach the MCP server. The script diff --git a/labs/local/troubleshooting.md b/labs/local/troubleshooting.md index 899487c..674c71a 100644 --- a/labs/local/troubleshooting.md +++ b/labs/local/troubleshooting.md @@ -13,14 +13,15 @@ Each failed line prints its fix. | Symptom | Fix | |---|---| -| `docker compose ... up` fails with a port already in use | Another program has one of the ports the stack publishes on `127.0.0.1`: 29092 (Kafka), 18081 (schema registry), 4566 and 5691 (RisingWave), 8000 (MCP). Stop that program. The ports are fixed: the broker tells its clients to come back to `127.0.0.1:29092`, and `local/write-env.sh` writes these ports into `.env`. | -| `local/engine.sh` fails because port 8080 is taken (`Bind for 0.0.0.0:8080 failed: port is already allocated`) | Pick another port for the registry: `export ORCA_LOCAL_REGISTRY_PORT=18080`, run `local/engine.sh` again, then `local/write-env.sh` so `.env` has the new address. Export it in every terminal you run `local/engine.sh` from: a run without it goes back to 8080. | +| `docker compose ... up` fails with a port already in use | Another program has one of the ports the stack publishes on `127.0.0.1`: 29092 (Kafka), 18081 (schema registry), 4566 and 5691 (RisingWave), 8000 (MCP). Stop that program, run `local/down.sh`, then run the `up` command again. Running it again without `local/down.sh` is not enough: Docker brings the container whose port was taken back without its network. The ports are fixed: the broker tells its clients to come back to `127.0.0.1:29092`, and `local/write-env.sh` writes these ports into `.env`. | +| `local/engine.sh` fails because a port is taken (`Bind for 0.0.0.0:8080 failed: port is already allocated`) | Another program has a port the engine publishes on `127.0.0.1`: 8080 (the registry) or 18082. If it is a container, `docker ps` shows which: look for `:8080->` under PORTS. Stop that program and run `local/engine.sh` again. If the port is 8080 and you want to keep that program running, move the registry instead: `export ORCA_LOCAL_REGISTRY_PORT=18080`, run `local/engine.sh` again, then `local/write-env.sh` so `.env` has the new address. Export it in every terminal you run `local/engine.sh` from: a run without it goes back to 8080. | | `local/engine.sh`: `ANTHROPIC_API_KEY is not set in this shell` | `export ANTHROPIC_API_KEY=` in the terminal where you run the script. The engine reads the key only when it starts. | | `local/engine.sh`: `bootstrap refused: an organization already exists` | The engine's volumes exist but its keys in `.lab/ork` are gone. Start over: `local/down.sh --reset`, then Lab 0. | +| CLI path: `.venv/bin/python: No such file or directory` | The Python path is not installed. If you installed the TypeScript path, use the `npm` command the lab gives beside the Python one. Otherwise install one of the two: Lab 0, "Before you start". | | Doctor: `Agent Engine HTTP 401` | The key in `.env` is not the running engine's key. Run `local/write-env.sh`. If it still fails, the engine's volumes and keys are out of step: `local/down.sh --reset`, then Lab 0. | | Doctor: `Kafka ... not found` | The topic does not exist yet: Lab 0, step 4. | | Doctor: `Schema Registry ... not found` | The schema is registered when you seed the topic: `python seed.py` or `npm run seed`. | -| Doctor: `Kafka`, `Schema Registry`, or `MCP server` cannot be reached, or `RisingWave through MCP` fails | The streaming stack is not up: `docker compose -f local/compose.yaml up -d --wait`. | +| Doctor: `Kafka`, `Schema Registry`, or `MCP server` cannot be reached, or `RisingWave through MCP` fails | The streaming stack is not up: `docker compose -f local/compose.yaml up -d --wait`. If it is up and the doctor still says so, a container lost its network when its port was taken: run `local/down.sh`, then start both stacks again. | | Doctor: `Agent Engine Connection error` | The engine is not up: `local/engine.sh`. Start the streaming stack first. | | `seed`: `already holds 246 events` | The topic is seeded. Nothing to do. | | `table or source not found: security.login_events` | Create the source: Lab 2, step 1. | diff --git a/local/engine.sh b/local/engine.sh index 0f512ef..0fce8bb 100755 --- a/local/engine.sh +++ b/local/engine.sh @@ -11,7 +11,9 @@ # 1. ork local --data-dir /.lab/ork start --with-gateway # The Agent Engine, with the AI Gateway. The gateway makes every MCP call on # your agent's behalf, so MCP tools need it. The data directory is given as a -# full path: ork v0.6.0 does not resolve a relative one. +# full path: ork v0.6.0 does not resolve a relative one. First, the engine's +# containers that are not running are removed, so that it starts from fresh +# ones. Its data is in volumes. # # 2. The link. The gateway refuses private MCP hosts unless they are on its # allowlist, and `ork local start` writes that allowlist empty every time. @@ -44,6 +46,14 @@ start_engine() { [ -n "$(mcp_container)" ] || die "The streaming stack is not running. Start it first: docker compose -f local/compose.yaml up -d --wait" + # Replace what is not running. A container whose port could not be bound + # (another program had it) stays cut off from its network: Docker starts it + # with loopback only from then on, even once the port is free (seen with + # Docker Engine 29.2). The engine's data is in volumes, so nothing is lost. + local name + while IFS= read -r name; do + [ -z "$name" ] || docker rm "$name" >/dev/null + done < <(engine_stopped_containers) ork local --data-dir "$ORK_DIR" start --with-gateway } diff --git a/local/lib.sh b/local/lib.sh index f831054..b473131 100644 --- a/local/lib.sh +++ b/local/lib.sh @@ -44,6 +44,12 @@ engine_container() { # engine_container --filter "label=com.docker.compose.service=$1" --format '{{.Names}}' | head -n 1 } +# The Agent Engine's containers that exist but are not running. +engine_stopped_containers() { + docker ps -a --filter "label=com.docker.compose.project.working_dir=$ORK_DIR" \ + --filter status=created --filter status=exited --filter status=dead --format '{{.Names}}' +} + mcp_container() { docker ps --filter "label=com.docker.compose.project=$STREAMING_PROJECT" \ --filter "label=com.docker.compose.service=risingwave-mcp" --format '{{.Names}}' | head -n 1 diff --git a/local/tests/run.sh b/local/tests/run.sh index 2b9f316..7bf4a2c 100755 --- a/local/tests/run.sh +++ b/local/tests/run.sh @@ -10,26 +10,53 @@ set -euo pipefail TESTS=$(cd "$(dirname "$0")" && pwd) LOCAL=$(dirname "$TESTS") WORK=$(mktemp -d "${TMPDIR:-/tmp}/hello-local-tests.XXXXXX") +WORK=$(cd "$WORK" && pwd) # as the scripts see it: a TMPDIR ending in "/" leaves a "//" trap 'rm -rf "$WORK"' EXIT -# A `docker` that knows which host port the registry is published on, and -# remembers what it was asked. +# A `docker` that knows which host port the registry is published on, which +# containers exist, and remembers what it was asked. mkdir -p "$WORK/bin" cat >"$WORK/bin/docker" <<'EOF' #!/usr/bin/env bash [ -n "${FAKE_DOCKER_DIR:-}" ] || exit 1 printf '%s\n' "$*" >>"$FAKE_DOCKER_DIR/calls" case "$1" in - ps) [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo registry-1 ;; + ps) + case " $* " in + *" -a "*) + # Every container, running or not: the " " lines of the + # fixture, when the question is narrowed to the engine's project. A + # status filter keeps the lines with that status. + if [[ " $* " != *" label=com.docker.compose.project.working_dir=${FAKE_DOCKER_DIR%/*}/.lab/ork "* ]]; then + echo someone-elses-container + elif [ -f "$FAKE_DOCKER_DIR/containers" ]; then + while read -r status name; do + [[ " $* " == *" status="* && " $* " != *" status=$status "* ]] || echo "$name" + done <"$FAKE_DOCKER_DIR/containers" + fi + ;; + *) [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo registry-1 ;; + esac + ;; port) [ ! -f "$FAKE_DOCKER_DIR/port-fails" ] || exit 1 [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo "127.0.0.1:$(cat "$FAKE_DOCKER_DIR/registry-port")" ;; + rm) [ $# -ge 2 ] || exit 1 ;; # like docker, it wants at least one container compose) ;; *) exit 1 ;; esac EOF chmod +x "$WORK/bin/docker" +# An `ork` that remembers the call and stops the script there: what comes after +# a start needs a running stack. +cat >"$WORK/bin/ork" <<'EOF' +#!/usr/bin/env bash +[ -n "${FAKE_DOCKER_DIR:-}" ] || exit 1 +printf 'ork %s\n' "$*" >>"$FAKE_DOCKER_DIR/calls" +exit 1 +EOF +chmod +x "$WORK/bin/ork" export PATH="$WORK/bin:$PATH" # Nothing from the developer's own shell may leak into the tests. unset ORCA_LOCAL_REGISTRY_PORT TUTORIAL_STACK PARTICIPANT ORCA_MODEL ORCA_API_KEY @@ -63,6 +90,7 @@ run() { # run