From b548f60a2a5bc490869f4591ebe2078d03fcfff7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Thu, 19 Nov 2026 23:04:05 -0500 Subject: [PATCH 1/2] A maintainer can find which system call the worker sandbox's filter killed (#27) docs/develop.md gains "A grant worker died of SIGSYS": the kernel audit line, tracing the Gateway under the runtime test with strace -D, mapping the number to a name per architecture, and where a new allow-list entry goes. Co-Authored-By: Claude Fable 5.2 --- docs/develop.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/develop.md b/docs/develop.md index 5c5002a..39bdbe9 100644 --- a/docs/develop.md +++ b/docs/develop.md @@ -27,6 +27,46 @@ pnpm test For the graphics and network path, run the [demo walkthrough](try.md) with `--verify` to automate the installed Companion and browser. The hosted end-to-end job also runs a separate harness that selects identities from candidate lists of different sizes, verifies the peer fingerprint and checks revocation. The native qualification jobs generate identities in disposable provider stores. On macOS and Windows use a disposable runner account for that harness because it changes user-level keychain or certificate-store state. [End-to-end workflow](https://github.com/zeroae/countersign/blob/fbb5aa7c951f8b70a238f3351140aa44a6184adf/.github/workflows/e2e.yml) ยท [Native qualification workflow](https://github.com/zeroae/countersign/blob/fbb5aa7c951f8b70a238f3351140aa44a6184adf/.github/workflows/native.yml) +## A grant worker died of SIGSYS + +A grant worker that makes a system call outside the worker sandbox's allow-list is killed by the kernel with `SIGSYS`. The Gateway logs `grant worker killed by the worker sandbox's filter (SIGSYS); grant fenced`, which does not say which call it was. Two commands do. + +The kernel writes an audit line for each filter kill. Reading it needs no root where your user can read the system journal, as members of `adm` or `systemd-journal` can. + +```sh +journalctl -k | grep type=1326 +``` + +```text +kernel: audit: type=1326 audit(1795144504.852:6342): ... pid=2188053 comm="worker_sandbox-" exe="..." sig=31 arch=c00000b7 syscall=56 compat=0 ip=0xec5e8e3f07c4 code=0x80000000 +``` + +`sig=31` is `SIGSYS`, `syscall=` is the number of the call and `arch=` the architecture that number belongs to: `c000003e` is x86-64 and `c00000b7` is arm64. In CI the same lines come from `sudo dmesg`, which the CI and end-to-end workflows print when a run fails, in the step "Name the system call when the worker sandbox's filter killed a grant worker". + +When the number is not enough, trace the Gateway. The runtime test takes the Gateway executable as its argument, so give it a wrapper that starts the real one under `strace`: + +```sh +cat > /tmp/traced-gateway <<'EOF' +#!/bin/sh +exec strace -D -f -o "/tmp/gateway-trace.$$" "$PWD/target/debug/countersign-gateway" "$@" +EOF +chmod +x /tmp/traced-gateway +python crates/countersign-gateway/tests/runtime.py /tmp/traced-gateway +``` + +The wrapper runs `strace -D -f -o target/debug/countersign-gateway serve ` for each Gateway the test starts, one trace file per Gateway. `-f` follows the Gateway into its grant workers. `-D` matters: the sidecar part of the runtime test stops every descendant of the Gateway, and without `-D` the tracer is one of them, so the Gateway it traces stops too and the test hangs. With `-D` the tracer detaches from the Gateway's process tree. In each trace file, search for `+++ killed by SIGSYS +++`. The line above it with the same process ID is the call the filter killed; ignore the result `strace` prints for it. Tracing slows the Gateway, so one of the test's timing assertions can fail under it. The trace up to that point is still complete. + +The same number is a different call on each architecture, so map it with the architecture from the audit line. `ausyscall`, from the `auditd` package, takes both: + +```sh +ausyscall aarch64 56 # openat +ausyscall x86_64 56 # clone +``` + +Without it, read the kernel's `unistd.h` for the architecture: `/usr/include/x86_64-linux-gnu/asm/unistd_64.h` on x86-64 and `/usr/include/asm-generic/unistd.h` on arm64, where `grep -w 56` finds `#define __NR_openat 56`. Read the header that matches the architecture of the killed worker, not of the machine you are reading on. + +A call a grant worker needs goes in the allow-list in `crates/countersign-gateway/src/worker_sandbox.rs`, with the reason beside it as a comment: `ALLOWED` for a call allowed whatever its arguments, or the argument-checked list below it. A call the worker should not make is a bug in the worker, not a gap in the list. + ## Install with conda on Linux On a Linux desktop with rattler-build and conda installed, place the local channel outside the checkout. Build, install and start the packaged application in the graphical login session. From 5dc63d9f2c1b1ae823fdba69f016eb058dca58b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patrick=20Sodr=C3=A9?= Date: Thu, 19 Nov 2026 23:10:15 -0500 Subject: [PATCH 2/2] The SIGSYS section names the right process tree for -D, and what the trace and the journal hold (#27) Review round 1: without -D the Gateway is the tracer's child, not the other way round; a filter kill ends every thread; the sample audit line is the containment test's; auditd takes the lines out of the journal; clone3 and argument-checked calls; the kernel's tables for a foreign architecture. Co-Authored-By: Claude Fable 5.2 --- docs/develop.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/develop.md b/docs/develop.md index 39bdbe9..61eabb7 100644 --- a/docs/develop.md +++ b/docs/develop.md @@ -29,7 +29,7 @@ For the graphics and network path, run the [demo walkthrough](try.md) with `--ve ## A grant worker died of SIGSYS -A grant worker that makes a system call outside the worker sandbox's allow-list is killed by the kernel with `SIGSYS`. The Gateway logs `grant worker killed by the worker sandbox's filter (SIGSYS); grant fenced`, which does not say which call it was. Two commands do. +The worker sandbox's filter kills a grant worker with `SIGSYS` when it makes a system call outside the allow-list, or a listed call with an argument the list refuses. One call is the exception: `clone3` fails as unsupported and kills nothing. The Gateway logs `grant worker killed by the worker sandbox's filter (SIGSYS); grant fenced`, which does not say which call it was. Two commands do. The kernel writes an audit line for each filter kill. Reading it needs no root where your user can read the system journal, as members of `adm` or `systemd-journal` can. @@ -38,12 +38,14 @@ journalctl -k | grep type=1326 ``` ```text -kernel: audit: type=1326 audit(1795144504.852:6342): ... pid=2188053 comm="worker_sandbox-" exe="..." sig=31 arch=c00000b7 syscall=56 compat=0 ip=0xec5e8e3f07c4 code=0x80000000 +kernel: audit: type=1326 audit(1795144504.852:6342): ... pid=2188053 comm="worker_sandbox-" exe=".../target/debug/deps/worker_sandbox-dcf0f669f7f13f3c" sig=31 arch=c00000b7 syscall=56 compat=0 ip=0xec5e8e3f07c4 code=0x80000000 ``` -`sig=31` is `SIGSYS`, `syscall=` is the number of the call and `arch=` the architecture that number belongs to: `c000003e` is x86-64 and `c00000b7` is arm64. In CI the same lines come from `sudo dmesg`, which the CI and end-to-end workflows print when a run fails, in the step "Name the system call when the worker sandbox's filter killed a grant worker". +`sig=31` is `SIGSYS`, `syscall=` is the number of the call and `arch=` the architecture that number belongs to: `c000003e` is x86-64 and `c00000b7` is arm64. The line above is not a grant worker's. It comes from the containment test in `crates/countersign-gateway/tests/worker_sandbox.rs`, which has the filter kill a process on purpose many times in each run, and `comm="worker_sandbox-"` marks every such line. Leave those out: a grant worker's line names another `comm` and `exe`. -When the number is not enough, trace the Gateway. The runtime test takes the Gateway executable as its argument, so give it a wrapper that starts the real one under `strace`: +In CI the same lines come from `sudo dmesg`, which the CI and end-to-end workflows print when a run fails, in the step "Name the system call when the worker sandbox's filter killed a grant worker". A host that runs the audit daemon, `auditd`, sends these lines to `/var/log/audit/audit.log` and keeps them out of the journal. + +When the number is not enough, trace the Gateway. The runtime test takes the Gateway executable as its argument, so give it a wrapper that starts the real one under `strace`. Besides `strace`, the test needs what its first lines list: bubblewrap, libnss-wrapper, curl and Python's `cryptography`. From the checkout root: ```sh cat > /tmp/traced-gateway <<'EOF' @@ -51,21 +53,23 @@ cat > /tmp/traced-gateway <<'EOF' exec strace -D -f -o "/tmp/gateway-trace.$$" "$PWD/target/debug/countersign-gateway" "$@" EOF chmod +x /tmp/traced-gateway -python crates/countersign-gateway/tests/runtime.py /tmp/traced-gateway +python3 crates/countersign-gateway/tests/runtime.py /tmp/traced-gateway ``` -The wrapper runs `strace -D -f -o target/debug/countersign-gateway serve ` for each Gateway the test starts, one trace file per Gateway. `-f` follows the Gateway into its grant workers. `-D` matters: the sidecar part of the runtime test stops every descendant of the Gateway, and without `-D` the tracer is one of them, so the Gateway it traces stops too and the test hangs. With `-D` the tracer detaches from the Gateway's process tree. In each trace file, search for `+++ killed by SIGSYS +++`. The line above it with the same process ID is the call the filter killed; ignore the result `strace` prints for it. Tracing slows the Gateway, so one of the test's timing assertions can fail under it. The trace up to that point is still complete. +The wrapper runs `strace -D -f -o target/debug/countersign-gateway serve ` for each Gateway the test starts, one trace file per Gateway. `-f` follows the Gateway into its grant workers. `-D` matters: the sidecar part of the runtime test stops every descendant of the process it started. Without `-D` that process is `strace` and the Gateway is its child, so the test stops the Gateway and hangs. With `-D` the Gateway is the process the test started, and the tracer runs outside its process tree. Tracing slows the Gateway, so one of the test's timing assertions can fail under it. The trace up to that point is still complete. + +In each trace file, search for `+++ killed by SIGSYS +++`. A filter kill ends every thread of the grant worker, so the file has one such line for each thread, and for most of them the line above is an allowed call the thread was waiting in. Look in the lines before the first of them for the call the audit line named. The trace shows its arguments and what the grant worker did before it; ignore the result `strace` prints for it. The last Gateway the test starts has one grant worker the test itself kills, marked `--- SIGSYS {si_signo=SIGSYS, si_code=SI_USER, ...} ---`. That one is not a filter kill. -The same number is a different call on each architecture, so map it with the architecture from the audit line. `ausyscall`, from the `auditd` package, takes both: +The same number is a different call on each architecture, so map it with the architecture from the audit line. `ausyscall` takes both: ```sh ausyscall aarch64 56 # openat ausyscall x86_64 56 # clone ``` -Without it, read the kernel's `unistd.h` for the architecture: `/usr/include/x86_64-linux-gnu/asm/unistd_64.h` on x86-64 and `/usr/include/asm-generic/unistd.h` on arm64, where `grep -w 56` finds `#define __NR_openat 56`. Read the header that matches the architecture of the killed worker, not of the machine you are reading on. +`ausyscall` comes in the `auditd` package, and installing that starts the audit daemon, which takes the audit lines out of the journal as above. Without it, read the kernel's table for the architecture of the killed grant worker, which need not be the machine you are reading on: [`syscall_64.tbl`](https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/syscall_64.tbl) for x86-64 and [`asm-generic/unistd.h`](https://github.com/torvalds/linux/blob/master/include/uapi/asm-generic/unistd.h) for arm64. A host of the same architecture has it installed, as `/usr/include/x86_64-linux-gnu/asm/unistd_64.h` on x86-64 and `/usr/include/asm-generic/unistd.h` on arm64, where `grep -w 56` finds `#define __NR_openat 56`. -A call a grant worker needs goes in the allow-list in `crates/countersign-gateway/src/worker_sandbox.rs`, with the reason beside it as a comment: `ALLOWED` for a call allowed whatever its arguments, or the argument-checked list below it. A call the worker should not make is a bug in the worker, not a gap in the list. +A call a grant worker needs goes in the allow-list in `crates/countersign-gateway/src/worker_sandbox.rs`, with the reason beside it as a comment: `ALLOWED` for a call allowed whatever its arguments, or `conditional`, in `allow_list`, for a call allowed only with certain arguments. When the killed call is already in `conditional`, an argument failed its test, and the trace shows which. A call a grant worker should not make is a bug in the grant worker, not a gap in the list. ## Install with conda on Linux