Repository navigation
fix(runner): an engine stopped by SIGTERM or SIGINT does not latch - #287
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An engine that something outside Berth stops with SIGTERM or SIGINT now shows as offline, not LATCHED (unresponsive).
The bug
On aphotic,
strata-sc117stayed LATCHED inberth statusand the harbor after a deliberate stop.strata stopin the homelab script sends SIGTERM to the Python server, which is the engineberth runstarted. The engine died of the signal, so the run saw exit code 143. The run only reportsFinishedto berthd on exit 0 or on a stop it carried out itself, so berthd read the exit as a crash and latched the engine. #277 covered a supervisor that signals theberth runwrapper, such as llama-swap. It did not cover a signal sent to the engine itself, askill PID, a stop script or Ctrl+C sends.The berthd log matches. The run that latched,
strata-sc117-11, released its lease at 16:39:33 with no drain before it, and the UD Strata started 20 s later.The fix
The run also reports
Finishedwhen the engine died of SIGTERM or SIGINT (exit code 143 or 130). Any other signal still latches, SIGKILL included, because that is what the OOM killer and a crash look like. So does a plain non-zero exit. On Windows nothing changes: there are no stop signals, and an exit code does not say who ended the engine.Proof
TestACleanExitDoesNotLatchTheEngine: SIGTERM, SIGINT and SIGKILL sent to the engine by itself. Onnext, SIGTERM and SIGINT fail withstate "latched", want "down". With the fix, all seven cases pass, and SIGKILL still latches.go build,gofmt,go vet,go test ./..., the Windows build and vet, and golangci-lint (0 issues) pass locally.