Skip to content

fix(runner): an engine stopped by SIGTERM or SIGINT does not latch - #287

Merged
T-Crypt merged 2 commits into
nextfrom
T-Crypt/sigterm-child-finished
Nov 2, 2026
Merged

T-Crypt merged 2 commits into
nextfrom
T-Crypt/sigterm-child-finished

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 2, 2026

Copy link
Copy Markdown
Owner

An engine that something outside Berth stops with SIGTERM or SIGINT now shows as offline, not LATCHED (unresponsive).

The bug

On aphotic, strata-sc117 stayed LATCHED in berth status and the harbor after a deliberate stop. strata stop in the homelab script sends SIGTERM to the Python server, which is the engine berth run started. The engine died of the signal, so the run saw exit code 143. The run only reports Finished to berthd on exit 0 or on a stop it carried out itself, so berthd read the exit as a crash and latched the engine. #277 covered a supervisor that signals the berth run wrapper, such as llama-swap. It did not cover a signal sent to the engine itself, as kill PID, a stop script or Ctrl+C sends.

The berthd log matches. The run that latched, strata-sc117-11, released its lease at 16:39:33 with no drain before it, and the UD Strata started 20 s later.

The fix

The run also reports Finished when the engine died of SIGTERM or SIGINT (exit code 143 or 130). Any other signal still latches, SIGKILL included, because that is what the OOM killer and a crash look like. So does a plain non-zero exit. On Windows nothing changes: there are no stop signals, and an exit code does not say who ended the engine.

Proof

  • The first commit adds three cases to TestACleanExitDoesNotLatchTheEngine: SIGTERM, SIGINT and SIGKILL sent to the engine by itself. On next, SIGTERM and SIGINT fail with state "latched", want "down". With the fix, all seven cases pass, and SIGKILL still latches.
  • Removing either signal from the check fails the test.
  • go build, gofmt, go vet, go test ./..., the Windows build and vet, and golangci-lint (0 issues) pass locally.

@T-Crypt
T-Crypt merged commit e82cdd0 into next Nov 2, 2026
2 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/sigterm-child-finished branch November 2, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant