feat(eventing): Phase 2 — make event signing deployable on a cluster - #884
Conversation
… keys #879 wired signing in both directions but left it unreachable outside a laptop: the ConfigMap shipped `*_VERIFY_KEYSET_PATH` and `*_SIGNING_KID` as empty strings and said seed paths "belong in the Deployment beside the volume" — and no Deployment had one. The only volume in either was `data: emptyDir`, so there was no way to get a key into a pod. That blocks S6/S8/S9, each of which is defined as "reproducible on a local Kind cluster". * `scripts/gen_signing_keys.py` — no key generation existed anywhere, so enabling signing meant hand-rolling `os.urandom(32).hex()` and deriving public keys in a REPL. Seeds are written 0600 under a 0700 directory and **never printed**; only kids and public keys go to stdout, so a pasted terminal transcript cannot leak a private key. Public keys are derived from the seeds rather than typed, so the keyset cannot disagree with the keys it authorizes. Refuses to overwrite an existing seed without `--force`, since that key may already be deployed and approved. * Both Deployments mount a seed Secret at `/etc/ce-signing` and the keyset ConfigMap at `/etc/ce-keyset`, both `optional: true` — without that, every signing-less overlay would block in ContainerCreating waiting for a Secret that will never exist. No `defaultMode`: 0400 would be root-only and there is no `runAsUser`/`fsGroup` (OpenShift assigns the UID), so the default 0644 is what makes the file readable. * `k8s/base/keyset-configmap.yaml` ships **empty** and a test will keep it that way. "There is never a key in this file" is checkable; "only public keys in this file" is not, and a 32-byte seed is indistinguishable from a public key by length. * `k8s/overlays/kind-signed/` flips enforcement on. A separate overlay rather than patching `kind`, because S3/S5 are explicitly unsigned milestones and the before/after contrast is the demo — and because the `test` overlay's "mounts no credential" assertion should keep meaning something. **A per-service Secret, not one shared.** Caught while rendering the overlay: both Deployments initially mounted the same `event-signing-key`, which would have given the two services one seed. That collapses the asymmetry the design rests on — EventBridge would hold the key it verifies runner responses with and could forge them, which is the exact property that made HMAC unacceptable (DESIGN_PHASE2.md §4.3). Now `eventbridge-signing-key` and `eventrunner-signing-key`. This deviates from the single `event-signing-key` name in README_PHASE1.md:704-722; that section is rewritten in a later commit, as it also still describes a symmetric setup. Verified: `data` remains volume index 0 in all three existing overlays, so the demo overlay's `/spec/template/spec/volumes/0` PVC patch still targets the right volume; no overlay renders a `kind: Secret`; `kind` stays fully off while `kind-signed` renders enforcement plus both key paths; the rendered env maps onto the real config fields; and keys from the generator verify through `verify_with_keyset`. 551 passed, 5 skipped — including the three manifest tests this could have broken (`test_no_credential_is_ever_rendered_into_a_manifest`, `test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic`, `test_no_manifest_sets_run_as_user`). ruff check and format clean. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…it on teardown `k8s_deploy.py --signing-keys DIR` applies one seed Secret per service plus the approved-key ConfigMap, modelled on `ensure_ntfy_secret`: via stdin so the seed never reaches `ps` through argv, and a no-op when the flag is absent so a redeploy cannot silently switch enforcement off and strand a signed topic. It validates before applying, because every one of these mistakes produces a cluster that refuses all traffic with no local symptom — the service starts, signs under a kid nobody approved, and the other side rejects everything: * the key directory and each named seed file must exist, * `agents.json` must parse as a keyset and approve at least one kid, * the kid each service will sign under must be IN that keyset. Each failure names the file and the command that fixes it. Seeds are logged as a length only — not even a prefix. 4 hex chars of a 32-byte seed is 16 bits of a private key, and unlike an API token there is nothing to gain from identifying it. The keyset gets the opposite treatment on purpose: kids are logged in full, because "the operator deployed yesterday's key set" is otherwise invisible until it surfaces as an unexplained rejection. `--overlay kind-signed` without `--signing-keys` now aborts unless both Secrets are already deployed. Both volumes are `optional: true`, so without this the pods would start happily with enforcement on and no key, and refuse every event. Teardown: `--purge` now deletes both signing Secrets and the keyset ConfigMap. It also deletes `eventbridge-ntfy`, which was listed in neither the `doomed` preview nor the delete tuple — so a "purged" namespace kept a capability that lets anyone who learns it read and publish your notifications. Leaving private keys behind is worse: a later deploy silently inherits an identity the operator thought they had destroyed. 17 new tests covering generation and application, including that the two services are never given the same seed (a shared one would let EventBridge forge any agent's response — the HMAC property §4.3 rejected), that seeds land 0600 under a 0700 directory, that the CLI prints public keys but never a seed on either stream, and that each malformed-key-directory case is refused before it reaches a cluster. 568 passed, 5 skipped. ruff check and format clean. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
aslom
left a comment
There was a problem hiding this comment.
Reviewed at 21d49c9 with design conformance as the primary lens. The diagnosis is
right and well evidenced — grep -rn "secretName\|configMap:" k8s/ returning nothing
against a ConfigMap that shipped *_KEYSET_PATH as empty strings is exactly the gap, and
configmap.yaml:54 really does promise a Deployment stanza that no Deployment had.
Design conformance: where it lines up
I checked every citation in the diff against the documents. They resolve, and the
reasoning is faithful:
| Claim | Document | Verdict |
|---|---|---|
| seeds → Secret, keyset → ConfigMap | DESIGN_PHASE2.md §5 |
✔ exactly as written, including why the location is the control |
| one seed per service, never shared | §4.3 — "EventBridge would hold the key it verifies with, so it could forge any agent's response" | ✔ the strongest argument in the PR, and it is the design's own |
| Ed25519, not ES256 | §4.3 | ✔ grep -rniE "es256" over agentdocs/ returns nothing — your read is right, only the issue commits to it, and §4.3 names SPIRE as the upgrade that leaves verification unchanged |
ER_REQUIRE_SIGNATURE, both *_KEYSET_PATH, both *_SIGNING_KID |
DESIGN_PHASE1.md §11 + DESIGN_PHASE2.md §5 |
✔ every name matches, and all six op: replace targets exist in the base |
EB_SIGNING_KID is the only kid accepted on group events |
§4.4 item 5 / §5 | ✔ the overlay comment restates it correctly |
bare §N = DESIGN_PHASE1.md |
pre-existing # §8.4 in eventrunner-deployment.yaml |
✔ consistent with the house convention |
Your three README claims are also accurate and, if anything, understated: :713 does say
event-signing-key, :710-716 does write a seed to /tmp and rm it against the
stdin-only rule at :474-477, and :715 does describe the symmetric setup §4.3 rejects.
It also says ER_VERIFY_KEY_PATH=/keys/seed.hex and mounts at /keys, so the mount path
drifts too.
One correction in your favour: the single-Secret convention you are deviating from is a
runbook, not a design. DESIGN_PHASE2.md §4.4 and §5 — the authoritative signing
design — say nothing about Secret names or count. The only design-level sentence nearby is
DESIGN_PHASE1.md §11's "Ed25519 keys in a read-only Secret", which two Secrets satisfy
as easily as one. So the deviation costs a README rewrite you already owe, not a design
amendment. Worth saying in the body so it does not read as a bigger decision than it is.
Where it does not line up
Three places, one of them with teeth.
-
The seed paths go into the ConfigMap, which
configmap.yaml:54-55says they must
not — and the overlay quotes that sentence immediately before doing it. Not a leak
(the value is a path), but it is a rule the repo states, the patch cites, and the patch
breaks. It also collides with #883, open right now: that PR adds
test_no_phase3_secret_is_inlined_into_a_manifest, assertingf"{var}:" not in renderedfor seed-path variables. ItsSECRET_PATH_VARSis scoped to Phase 3's three,
so it does not fail today — but §8.3 states the rule as "continuing Phase 2 §5's", so
ER_/EB_SIGNING_KEY_PATHjoining that tuple is the obvious next edit, and it would
fail against this overlay. Inline comment with both options. -
test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automaticnow passes for
a narrower reason than it documents. Its docstring says "NOTHING can inject a
credential into these pods: no Secret is referenced" — and thetestoverlay now
renderssecretName: eventrunner-signing-key. Only the literalsecretRefspelling in
the assertion keeps it green. You cite this test as a constraint that shaped the design,
so it is worth keeping it able to mean that. -
DESIGN_PHASE1.md§12's preflight table documents 12 checks; the script has 13
(check_claude_cli_image), whichIMPLEMENTATION_REPORT1.md:360,776and
README_PHASE1.md:189all reference by number. §13 step 0 also still reads "checks 1-8,
10-12". Pre-existing drift, not yours — but you are about to add check 14, so the
§12 table and the §13 step list are two more lines for the same rewrite pass. Worth
folding in rather than discovering later that the design documents 12 of 14.
Also: /etc/ce-signing, /etc/ce-keyset, seed.hex, eventing-keyset and the two
Secret names appear in no document in agentdocs/. That is fine for a draft, and your
README rewrite covers the runbook half — but note the feature's design lives in
DESIGN_PHASE2.md §4 while the rewrite target is README_PHASE1.md, and README_PHASE2.md
is already owed (it came up on #883). Putting the signed-deployment runbook into the Phase 1
document is the kind of choice that is cheap now and confusing in a month.
The must-fix
ensure_signing_material validates everything except the one thing that matters. It
checks the directory, that agents.json parses, that it approves at least one kid, that
each seed file exists, and that the kid is in the keyset — but never that the seed
derives the public key the keyset holds for that kid. A divergent pair passes every
check and produces precisely the cluster the function exists to prevent: pods start, both
volumes are optional: true, enforcement is on, and every event is refused with nothing
local to look at.
That state is reachable without hand-editing, which is the second finding: write_seeds
writes kid-by-kid and raises mid-loop on an existing seed, while agents.json is only
rewritten after it returns. Both reproduced below. Three lines in the deploy script
closes the whole class however the divergence arose — including the likeliest real case,
an agents.json copied from a teammate while the seeds are local.
Verification I ran
| Check | Result |
|---|---|
python -m pytest in eventing/ @ 21d49c9 |
566 passed, 7 skipped (573 total — your 568/5 agrees; 2 extra skips are this sandbox's subprocess block) |
17 new tests in test_signing_deploy.py |
confirmed |
ruff check . / ruff format --check . @ 0.11.4 |
clean, 135 files formatted |
kustomize build on all four overlays |
all four build |
data is volume index 0 everywhere |
✔ ['data', 'signing-key', 'keyset'] in all four; demo's index 0 is the PVC, so its op: replace still lands on data |
no overlay renders kind: Secret |
✔ |
kind-signed renders enforcement + both key paths |
✔ all eight values, and they map onto real Cfg fields |
kind stays fully off |
✔ both flags "false", all four paths empty, no *_SIGNING_KEY_PATH key at all |
| CI | 12/12 green |
| Commits | 2, both Signed-off-by, Assisted-By not Co-Authored-By |
.claude/ / .vscode/ changes |
none |
I also applied both code suggestions to a scratch worktree: 566 passed, 7 skipped,
ruff still clean, and three tests written against the fixed behaviour pass — a
mismatched pair is now refused with nothing applied, a matching directory still applies
all three objects, and a refused regenerate leaves the directory byte-identical. Each
fails against its own revert.
kustomize build succeeding is weaker than kubectl apply --dry-run=server, which I
also cannot run — so your "schema validity of the new volume stanzas is unproven" stands.
It does rule out the failure mode I most expected: a replace op against a key that does
not exist in the base.
On your question
kind-signed as a separate overlay is the right shape. Two reasons beyond the ones
you give. A --signed flag on kind would have to mutate the ConfigMap at deploy time,
which puts enforcement outside the §14.1 manifest digest — the thing you correctly argue
the values belong inside. And the forged-event demo (S9) wants both overlays deployable
back to back in one session, which a flag on one overlay makes awkward. The name is fine;
kind-signed reads as "kind, plus signing" and that is what it is.
REQUEST_CHANGES for the missing seed/keyset agreement check only. Everything else is a
suggestion or a note on the rewrite you have already scheduled.
…d three more Addresses the #884 review. All findings reproduced against `21d49c9` before fixing, and each fix verified by reverting it. **The must-fix: `ensure_signing_material` validated everything except the one thing that mattered.** It checked the directory, that `agents.json` parses, that it approves a kid, that the seed file exists, and that the kid is *in* the keyset — never that the seed **derives** the public key the keyset holds for it. Reproduced: kid in keyset : True seed file exists : True keyset parses : True pub(seed) : 29acbae141bccaf0… keyset's pub for eb-01 : 03a107bff3ce10be… MATCH : False A divergent pair passed every check and produced exactly the cluster the function exists to prevent: pods start (both volumes are `optional: true`), enforcement is on, the service signs with a key nobody approved, and every event is refused with nothing local to look at. Now refused before anything is applied, with both fingerprints and the two remedies named. A corrupt seed is a named check failure rather than a traceback. **`write_seeds` is all-or-nothing.** It wrote kid-by-kid and raised mid-loop, while `agents.json` is rewritten by the caller afterwards — so if the FIRST kid in sorted order was missing and a later one present, the first got a brand-new seed and the keyset kept the old public key for it. That is the one state the module docstring promises cannot happen ("the keyset cannot disagree with the keys it is supposed to authorize"), and it is reachable from a partial rotation or a restored backup. The pre-check now runs over the whole set before any file is touched, and the refusal says "Nothing was written". Both of these are the same hazard from opposite ends, which is why the deploy-time check matters even with the generator fixed: the likeliest real case is neither, but an `agents.json` copied from a teammate while the seeds on disk are local. **The seed paths move out of the ConfigMap and into each Deployment's `env`.** The overlay quoted `configmap.yaml:54-55` — "they name Secret mounts and belong in the Deployment beside the volume" — immediately before doing the other thing. No leak, since the value is a path, but a rule the repo states and the patch cites should not be a rule the patch breaks. Followed the rule rather than rewriting it, which also keeps `test_manifests.py`'s Secret-vs-ConfigMap assertions aligned and leaves #883 free to extend its `SECRET_PATH_VARS` tuple to these two variables, as §8.3 implies it should. `env` does not exist on either base container (only `envFrom`), so the patch adds the whole list; an `env/-` append would fail at render time. All four overlays still build, `kind` stays fully unsigned, and `kind-signed` renders both paths on the Deployments with enforcement still in the ConfigMap where the §14.1 digest covers it. **`test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic` passed for a narrower reason than it documented.** Its docstring says "no Secret is referenced", and the overlay now renders `secretName: eventrunner-signing-key` — only the literal `secretRef` spelling kept it green. It now checks Secret references by name against an allowlist of the two signing Secrets and rejects anything `anthropic`-shaped by either spelling. Verified by injecting a credential Secret into the test overlay: the test goes red, where before it would not have. **§12 documented 12 preflight checks; the script has 13.** `check_claude_cli_image` was added with the demo overlay and never got a table row, while `IMPLEMENTATION_REPORT1.md` and `README_PHASE1.md` both reference checks by number. Added row 13 with its reasoning (it runs on a cluster node because the CLI's bundled `bun` aborts under QEMU user-mode emulation), and corrected "checks 1-8, 10-12" to "10-13" in §13 step 0 and both places `k8s_deploy.py` repeats it. Pre-existing drift, folded in here because check 14 is on this branch's own list. Tests: 575 passing (+7), ruff clean, all four overlays build. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
|
All four findings addressed in Your correction, accepted and now in the body. You are right that the single-Secret §12/§13 preflight drift, folded in. You were right that this is pre-existing and On the runbook target, which I think you are right about and I want to flag rather than So the plan is That makes the remaining work on this branch:
On On ES256 vs Ed25519 — your Tests: 575 passing (+7), ruff clean, all four overlays build. |
aslom
left a comment
There was a problem hiding this comment.
Verified at 803ccd9. All four round-1 findings are fixed, and I checked each against
the tree rather than against the replies — including mutation-reverting every fix.
| # | Finding | Fix | Verified by |
|---|---|---|---|
| 1 | ensure_signing_material never checked pub(seed) == ks[kid] |
the check, with both remedies in the message | mutation: 4 failures; my own round-1 repro now refuses the mismatched pair with nothing applied |
| 2 | write_seeds could leave a new seed beside the old public key |
whole-set pre-check, "Nothing was written" | mutation: 2 failures (one more than you claimed); ran the first-kid-missing case — seed not re-created, agents.json untouched |
| 3 | seed paths in the ConfigMap contradicted configmap.yaml:54-55 |
moved to container env on both Deployments |
rendered all four overlays: no overlay has a *_SIGNING_KEY_PATH key in eventing-config, and kind-signed carries both on the Deployments |
| 4 | the test-overlay assertion passed for a narrower reason than it documented |
secretName: allowlist by name, plus an anthropic-shaped reject |
read; see the note below on what exercises it |
Two things you did beyond what I asked, both right:
load_seedrather thanbytes.fromhex, so a corrupt or wrong-length seed is a
named check failure instead of a traceback — and it means a public key accidentally
placed inseed.hexis caught by the derivation check rather than silently decoding.DESIGN_PHASE1.md§12 now documents check 13 and §13 step 0 reads "checks 1-8,
10-13". That was in my review body with no commentable line, so thank you for picking
it up from prose.
Your reasoning on finding 3 is also the one I'd have argued for: following the rule costs
one extra patch block, while changing it would have meant coordinating a carve-out across
two open PRs and three statements of the same sentence. And you are right that #883
needs no change as a result — I re-checked, its SECRET_PATH_VARS tuple can be extended
to ER_/EB_SIGNING_KEY_PATH freely now and this overlay stays green.
Verification I ran
| Check | Result |
|---|---|
python -m pytest in eventing/ @ 803ccd9 |
573 passed, 7 skipped (was 566/7) |
ruff check . / ruff format --check . @ 0.11.4 |
clean, 135 files formatted |
kustomize build on all four overlays |
all four build |
Seed-path keys in any rendered eventing-config |
none, in all four |
kind-signed Deployment env |
EB_SIGNING_KEY_PATH / ER_SIGNING_KEY_PATH, enforcement still in the ConfigMap where §14.1's digest covers it |
kind still fully unsigned |
both flags "false", all paths empty, no env entries |
demo's three env entries intact |
yes — see the one nit below, which is about a future overlay, not this one |
| My round-1 fixed-behaviour tests | 3/3 pass |
New tests in test_signing_deploy.py |
17 → 23 |
| Commits | 3, all Signed-off-by, Assisted-By not Co-Authored-By |
.claude/ / .vscode/ changes |
none |
Ready to merge? The code is. CI is not — and it is not your fault
Almost nothing has actually run on 803ccd9, and what shows as failed was cancelled
rather than failing. Job-level detail from the Security Scans run (37366549985):
trivy-scan cancelled 19:55:44Z -> 20:27:32Z failed steps: (none)
hadolint cancelled 19:55:44Z -> 20:28:09Z failed steps: (none)
codeql cancelled 19:55:44Z -> 20:30:12Z failed steps: (none)
dependency-review success 20:08:51Z -> 20:08:58Z
Three jobs sat queued for ~32 minutes and were cancelled with zero failed steps — they
never executed. dependency-review ran as soon as a runner freed up and passed in 7 s. The
whole CI workflow (lint, test, eventing-test, test-startup) is still queued
~35 minutes after creation, and PR Verifier recorded startup_failure twice. hadolint
and trivy-scan passed on 21d49c9 in 7 s and 33 s, and this PR touches no Dockerfile and
adds no dependency.
So the gate is unresolved, not red: this is runner starvation, not a result. Re-trigger the runs and confirm 12/12 before
merging — an empty push or closing and reopening the draft will do it. If eventing-test
comes back failing after a genuine run, that would be new information and worth a look,
but locally it is 573 passed with the same pytest and kafka-python CI installs.
One more gating item, unrelated to CI: this is still a draft, so the merge decision is
yours either way. If you want the remaining four items on your own list (preflight check
14, the e2e assertions, --forge, the README_PHASE1 rewrite) in this PR rather than a
follow-up, say so and I will hold the approval for them; I would rather see them land
separately with their own tests.
Approving the code. Both must-fix mechanisms are closed and reproduced closed, the
three suggestions are implemented, the design-conformance contradiction is resolved in the
direction that keeps all three statements of the rule aligned, and nothing regressed. The
nit below is about an overlay that does not exist yet.
| # `env` does not exist on the base container (only `envFrom`), so this adds the | ||
| # whole list rather than appending to it. A trailing-slash `env/-` op would fail | ||
| # with "path does not exist" at render time. |
There was a problem hiding this comment.
[nit] This comment warns about the failure that is loud and not about the one that is
silent — which matters because the PR plans to copy this patch into three more files.
env/- failing with "path does not exist" is correct and you are right to record it. The
hazard worth the comment is the other direction: op: add on /…/containers/0/env
replaces the list when one already exists. It is safe here because kind-signed
builds on ../kind, whose containers have no env — I rendered all four overlays to
confirm:
kind eventrunner env=[]
kind-signed eventrunner env=[('ER_SIGNING_KEY_PATH', '/etc/ce-signing/seed.hex')]
demo eventrunner env=[('ER_MOCK_CLAUDE','false'), ('HOME','/data/home'),
('CLAUDE_CONFIG_DIR','/data/home/.claude')]
demo is the one that has entries. I built a throwaway demo-signed overlay by changing
nothing but the base — - ../kind → - ../demo — with this patch block verbatim:
hypothetical demo-signed eventrunner env = ['ER_SIGNING_KEY_PATH']
demo's own = ['ER_MOCK_CLAUDE', 'HOME', 'CLAUDE_CONFIG_DIR']
-> demo's three entries are LOST, silently
kustomize build succeeds, nothing warns, and ER_MOCK_CLAUDE=false is gone — so a
demo-signed built the obvious way would quietly run the demo in mock mode, which is
precisely what test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic
exists to keep honest at the other end.
Not a defect in this PR, and I am not asking for a demo-signed overlay. Just the half-
sentence, since whoever writes it will start from this file:
| # `env` does not exist on the base container (only `envFrom`), so this adds the | |
| # whole list rather than appending to it. A trailing-slash `env/-` op would fail | |
| # with "path does not exist" at render time. | |
| # `env` does not exist on the base container (only `envFrom`), so this adds the | |
| # whole list rather than appending to it. A trailing-slash `env/-` op would fail | |
| # with "path does not exist" at render time. | |
| # | |
| # NOTE for anyone basing a signed overlay on `demo` instead: `add` on this path | |
| # REPLACES an existing list rather than extending it, and silently — `demo`'s | |
| # container has three entries (`ER_MOCK_CLAUDE=false`, `HOME`, | |
| # `CLAUDE_CONFIG_DIR`), and a copy of this block drops all three while | |
| # `kustomize build` still succeeds. `ER_MOCK_CLAUDE=false` going missing puts the | |
| # demo back in mock mode. From `demo`, append with `op: add` on `env/-` per entry. |
| allowed_secrets = {"eventbridge-signing-key", "eventrunner-signing-key"} | ||
| referenced = set(re.findall(r"secretName:\s*(\S+)", doc)) | ||
| assert referenced <= allowed_secrets, \ | ||
| f"unexpected Secret reference: {sorted(referenced - allowed_secrets)}" | ||
| # No credential Secret, by either spelling, and nothing credential-shaped. |
There was a problem hiding this comment.
[nit] This is the right assertion and nothing automated runs it.
The allowlist-by-name is what I was asking for — secretName: only, so the signing-key
volume name is not mistaken for a Secret reference, and that trap is worth the comment
you put on it. I could not mutation-verify it, though: the test is @needs_kubectl and
skips without a cluster, so it is 1 of the 5 no reachable cluster skips both here and in
CI. Reverting the three new lines leaves tests/test_manifests.py at 29 passed, 5 skipped
— unchanged.
Your manual check (patching a credential Secret into the test overlay and watching it go
red) is the right verification and I believe it; I am noting the gap rather than disputing
the fix, because this is the same shape as IMPLEMENTATION_REPORT3.md §2's note about the
claude --help skip: a promise that currently only holds on a laptop.
render() is the thing that needs kubectl, and kustomize is a separate binary that
this sandbox has — so if render() could fall back to kustomize build when kubectl is
absent, this test and the four other manifest assertions would start running in CI with no
cluster. That is a bigger change than this PR should carry; worth an issue alongside
#885-#889 so it is not rediscovered the next time a manifest assertion silently stops
meaning anything.
…and how to check them (#890) * docs(eventing): record the claims a docs review disproved, and how to check them The user-facing eventing pages (rossoctl/rossoctl#2609) restated every claim in DESIGN_PHASE2 for a reader who cannot see the code. Six review rounds then ran each one against eventing/ at d9677dd. Thirteen did not hold. DESIGN_PHASE2 §8 records them, rather than editing §§2.6, 3, 4.3 and 4.4 in place, so the reasoning stays readable as what was intended and §8 says what the code does. The four §4.4 promises that fail: audit mode logs nothing, a pinned group event is flagged and then applied anyway, a terminal signature stops proving who finished a run once INSERT OR REPLACE lets an unsigned frame overwrite it, and nothing checks that EB_SIGNING_KID is in the keyset. A fifth, promised nowhere and worse: EB_REQUIRE_RESPONSE_SIGNATURE with no keyset refuses nothing, silently, while the request side with no key refuses everything. CLAIMS.md turns that into five questions to ask of a control claim before it ships — which code path, what default, what happens half-configured, can an attacker choose the input the check reads, what does it become one release from now — plus how to run the check cheaply and what a correction should preserve. None of the thirteen was careless about mechanism; §4.4 is unusually precise. Each was written from the change that introduced the control and stayed true for the configuration that change was exercised in. Every one coexisted with a passing test suite. Tracked as #885, #886, #887 and #888; #889 came out of the same review without being a claim in any document. Docs only. No code changes, so this does not overlap #884, and it touches agentdocs/README.md where #883 also does — whichever merges second resolves one table row and one reading-order entry. Assisted-By: Claude Code Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * docs(eventing): fold CLAIMS.md into DESIGN_PHASE2.md §8, per review @aslom on #890: "please merge CLAIMS into DESIGN_PHASE2.md or DESIGN_PHASE3.md as I think the claims are part of design and they should reflect the phase of the design they are related to." Taken. `CLAIMS.md` is gone and its content is now §8.5–§8.8 of `DESIGN_PHASE2.md`, directly after the thirteen findings it was derived from. Phase 2 is the right home: the findings are claims *this* document made, and §8.4 already handed off to the procedure, so the two halves were a section boundary apart for no reason. The content moved rather than being rewritten — 133 lines out, 135 in. Headings demote one level (the five questions become §8.5.1–§8.5.5) and the three references that assumed a separate file are re-pointed: "see its §8 for the list" is now "see above", and "`DESIGN_PHASE2.md` §8 is that shape" is now "§8.1–§8.3 above are that shape". **On the phase question, which is the substance of the request.** The findings are Phase 2's and the procedure is not, so §8 now opens by saying which is which: > §8.1–§8.4 are about **Phase 2** specifically: thirteen claims *this document* made > about the identity and signing controls, which do not hold in the code. §8.5–§8.8 are > the general procedure that came out of them, and apply to a claim made in any phase — > Phase 3's design makes claims of the same shape about tenancy and the signed attribute > set, and nothing here is Phase 2 only. That answers the question the PR body asked out loud ("is `CLAIMS.md` the right home? It is process guidance sitting among design and build records") rather than leaving the ambiguity in place. §8.5's own lead-in says the same thing for a reader who arrives there directly, and gives the reason it is filed here rather than at repo root: a procedure kept away from the findings that motivated it is a procedure nobody reads. `agentdocs/README.md`: the standalone index row is gone, the Phase 2 row now mentions both halves of §8, and the reading-order entry reads "**Writing a claim about a control, in any phase?**" pointing at §8.5 for the questions and §8.1–§8.3 for the findings — with "The findings are Phase 2's; the questions are not" stated explicitly, since that distinction is the whole point of where this now lives. Verified every `§N.N` reference in the merged document resolves to a heading that exists, and that no `CLAIMS` reference survives anywhere. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> --------- Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…kind-signed Follow-up to rebasing onto `main` now that #884 and #890 have merged. Both predicted conflicts were in `agentdocs/README.md` and resolved as planned — `main`'s Phase 2 row (which gained §8 from #890) plus this branch's Phase 3 row. `test_manifests.py` auto-merged, keeping #884's `allowed_secrets` fix beside this branch's Phase 3 classification. That left two things the merge did not resolve by itself. **Extending `SECRET_PATH_VARS` to #884's seed paths needed the rule split, not widened.** #884's review predicted this tuple would be extended to `ER_/EB_SIGNING_KEY_PATH` and checked that doing so stays green. It does for the ConfigMap half and would NOT for the literal-value half, because those two variables hold a **mount point, not key material** — and #884 deliberately sets them as container `env` entries so the path sits beside the volume that supplies the seed, which is what `configmap.yaml` asks for. So the tuple is now two: - `SECRET_VALUE_VARS` — the three whose *value* is key material. No literal value anywhere, which is the original assertion. - `SEED_PATH_VARS` — #884's two. A literal value is correct; a ConfigMap key is not. Both still obey "never from `config.toml`", so that check covers all five now where it covered three. Treating a path naming a secret as if it were the secret would have forced #884 to either revert its own correct fix or carve itself out of the rule — the distinction is the thing worth encoding. **`kind-signed` was not in `OVERLAYS`.** #884 added the overlay and every assertion in this file silently skipped it — including the seed-path rule the PR exists to respect, the `replicas` invariants, and the no-credential checks. Added, which is also what makes the new assertion meaningful: with `kind-signed` absent, injecting a seed path back into `eventing-config` passed; with it present, that regression fails as it should. Verified both ways. Tests: 899 passing (875 here + #884's 24 picked up in the rebase), ruff clean, all four overlays build. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
The remaining nit from the #884 approval, which merged before it was addressed. My comment on both patch blocks warned about the failure that is **loud** — an `env/-` append failing with "path does not exist" when no list exists — and not about the one that is **silent**: `op: add` on `…/containers/0/env` *replaces* the list when one already exists. Safe in `kind-signed`, which builds on `../kind` whose containers have no `env`. Not safe in the overlay the PR plans to copy this patch into. Reproduced by building a throwaway `demo-signed` that changes nothing but the base: demo-signed eventrunner env = ['ER_SIGNING_KEY_PATH'] demo's own = ['ER_MOCK_CLAUDE', 'HOME', 'CLAUDE_CONFIG_DIR'] -> all three LOST, silently `kustomize build` succeeds and nothing warns. Losing `ER_MOCK_CLAUDE=false` means a `demo-signed` overlay would run the demo in **mock mode** — which is exactly what `test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic` exists to keep honest at the other end, so the failure would be invisible from both directions. The comment now says that, and names the fix for whoever writes that overlay: `op: add` on `env/-` per entry, against a base that already has the key. Reviewer's words, and the right call to spend half a sentence on — this file is the starting point for three more (preflight, e2e, demo flow) by the PR's own plan. 575 passing, `kind-signed` still builds, no behaviour change. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…s, declarative agents (#883) * feat(eventing): Phase 3 step 1 — tenancy keys and declarative agent specs DESIGN_PHASE3.md T1 and T13, the two tasks with no dependencies. Nothing here changes behaviour: `single` tenancy is the default everywhere and the `default` AgentSpec reproduces Phase 2's argv byte for byte. shared/tenancy.py (T1) `userkey(issuer, userid)` is the only place an identity becomes a name. The slug is deliberately lossy so keys stay readable in `kubectl get kafkatopics`; that is safe only because 32 bits of SHA-256 over the canonical form are appended, so two identities that slug identically still differ. The issuer is inside the hashed bytes as well as the prefix — without it, configuring a second issuer is a way to impersonate an identity on the first. The digest input is joined with \x1f so the issuer/userid boundary is unambiguous. Canonicalisation is per-issuer and minimal: GitHub folds case (and must agree with `ghauth.is_allowed`, which a test pins), email lower-cases only the domain per RFC 5321, static is byte-exact. Erring toward two tenants for one human is wasteful; the other direction is a leak. `TopicSet` is the one place §3.1's layout is written down, and refuses to name a topic without a userkey in multi mode rather than falling back to a shared one. `ntfy_topic` is HMAC-derived so the name carries no identifier (§4.1). eventrunner/agentspec.py (T13) One TOML file per agent, carrying the tool policy that §7.4 depends on — which is why §9 puts this before triggers. Validation refuses rather than coerces: a policy that silently drops the clause it could not parse would hand a trigger-driven agent the tool the operator meant to remove. A missing non-default agent is an error event, never a silent fallback to `default`. `build_cmd` takes a spec. The request still wins on `max_turns`/`model` (Phase 0 wire parameters), but the tool policy, permission mode and system prompt are spec-only: a request that could widen the sandbox is a request that can escape it. Tests: 105 new, 656 total passing, no existing test modified. The argv-equality test compares against a literal transcription of Phase 2's `build_cmd` rather than a call into the current one, so it catches a regression instead of following one. The `claude --help` flag assertion covers only the flags Phase 3 adds. `--max-turns` is supported but undocumented in `--help` on 2.1.270, so asserting the pre-existing flags would fail against a working deployment. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * feat(eventing): Phase 3 steps 2-4 — TopicSet wiring, user registry, signed tenancy DESIGN_PHASE3.md T2, T3, T4 and T22. Still no behaviour change in the default configuration: `EB_TENANCY_MODE=single` is the default, and the 656 tests that existed before this branch all pass untouched. That is the property §9 asks steps 1-4 to preserve — refactors with tests, everything risky behind a flag that is off. T2 — TopicSet threaded through (§3.2) One set, built once in `__main__` and handed to the producer, the responses consumer, both mirrors and the config, so §3.1's layout appears exactly once. `publish_request` takes a `userkey` and resolves the topic from it; one producer can send to any topic, so there is no second producer and no connection per user. The explicit-topic-list option, not `subscribe(pattern=...)`. A pattern is picked up by metadata refresh (5 min by default), so a new user's first response can be published before anyone is subscribed and with `auto_offset_reset=latest` it is never read — the page stays empty and no error appears anywhere. The blocking `ensure_subscribed` that closes the remaining race is T6. GroupMirror takes the full topic list directly: it is one-shot at startup, so there is no new-user race to lose, and a tenant whose topic is not provisioned yet is skipped with a log line rather than costing everyone their group history. T3 — the registry and Caller (§2.4, §2.5) `Caller` carries the tenancy key so no caller derives it twice. `submitter_iss` is deliberately not the same as `issuer`: a static identity needs an issuer for its key to separate from a GitHub login of the same name, but Phase 2 made an absent `ce_submitteriss` mean "an operator typed this", and promoting it would upgrade an assertion into a verified claim. The registry refuses rather than skips — the opposite of `auth.parse_tokens`, because a skipped user is one whose topics exist and whose events have nowhere to go. A recorded userkey that disagrees with this build's derivation is a startup refusal naming both values, and two identities colliding on one key is refused, which is what covers §2.3's 32-bit birthday bound. `multi` mode refuses anonymous with 401 and an unregistered user with 403. Both disagree with the single-tenant defaults on purpose; `single` is byte-identical to Phase 2. T4 — SIGNED_ATTRS += userkey, depth (§8.3) One change, both attributes, because adding one changes canonicalisation and a signer and verifier on different versions disagree about every signature. Both have to be covered rather than merely present: a mutable `userkey` lets anything with topic write access file events into another user's history, and a resettable `depth` is not a hop limit. T22 — the Secret-vs-ConfigMap rule, pinned in test_manifests.py The secret-path variables are declared ahead of the tasks that read them (T8-T16) so the guard is in place when they land. The classification test only asserts the ConfigMap side against live code, and says so, rather than looking stronger than it is. Known gap, stated rather than hidden: `/continue` resolves the owning tenant from the recorded submitter so a resume reaches the right runner, but does not yet check that the caller IS the owner — owner-scoped reads and `?k=` keys are §6.2/§4.4 (T7/T9, step 5). Until those land, `multi` mode's `/continue` is as open as Phase 2's. The lookup also cannot distinguish a GitHub `alice` from a static `alice`; `multi` mode refuses both anonymous and static submissions, so GitHub is the only issuer that reaches it today, and this moves to T5's global index when that exists. Tests: 66 new (722 total), ruff clean. Verified both services start in every tenancy configuration and that all four refusals print readable messages rather than tracebacks. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * docs(eventing): record which of Phase 3 is implemented and which is still design The Phase 3 design doc said "nothing in this document is implemented yet" and the agentdocs index said "Design only — not implemented". Both are now wrong in a way that matters: a reader who trusts them will either re-implement steps 1-4 or assume `EB_TENANCY_MODE=multi` gives them isolation it does not yet give. So the status lines name the tasks that landed (T1, T2, T3, T4, T13, T22 = §9 steps 1-4) and say plainly that everything from step 5 on — per-user stores, owner-scoped reads, transcript auth, `k8s_tenant.py`, ntfy isolation, triggers, fetched skills, Kafka ACLs — is still design only. The component README gains a Phase 3 configuration table covering only the six variables whose code exists, with two notes that are the point of including it at all: `multi` mode is not yet isolation anyone should present as such (the HTTP reads and `/continue` are still as open as Phase 2's, because steps 5-8 are not in), and adding a user is an operator action because EventBridge has no Kubernetes client. Verified against the code rather than assumed: only `POST /v0/agents` and `POST /v0/groups` call the auth path, so every GET, `/continue` and `/transcript` is open in both modes today. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): refuse an unresolvable /continue with 503 instead of crashing Found reviewing the previous commit. In `multi` mode `_owner_userkey` returned `None` both when no key was needed (single mode) and when the owning tenant could not be determined. The second `None` flowed into `publish_request` -> `TopicSet.requests(None)`, which raises `ValueError` by design — publishing a resume to a shared topic would run it on another tenant's runner, under that tenant's credential — but nothing caught it, so the request died as a WSGI 500 with a stack trace instead of a status code. Three ways to reach it, none of them abuse: * a user removed from the registry; * a `prompts` row whose `submitter` is NULL — the column is nullable and pre-dates auth, so these rows exist in any store that outlived Phase 2; * a correlation back-filled from the topic by `RequestsMirror`, which records no submitter at all. `_owner_userkey` now returns `(userkey, unresolved_reason)` with exactly one set, so the two `None` cases are distinguishable, and both `/continue` paths check it before writing anything — no touched session row and no orphan prompt row claiming a turn that never ran. The JSON route answers `503` naming what could not be resolved, matching how §2.5 treats a user whose topics do not exist. The HTML form route answers `503` with a plain-text body rather than redirecting: a silent `303` back to the transcript page would look like the turn was accepted and then vanished, which is the Phase 2 §6.1 symptom class this phase keeps trying to avoid. Only reachable with `EB_TENANCY_MODE=multi`, so the default path was never affected. 13 new tests (735 total). Verified they are load-bearing by reintroducing the bug: the four refusal tests fail without the guard and pass with it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * feat(eventing): Phase 3 T5 — per-user stores and the global ownership index DESIGN_PHASE3.md T5 (§6.1, §2.6), the keystone the rest of step 5+ depends on: T7 (owner-scoped reads), T8 (transcript auth), T15 (triggers) and T21 (deletion) are all blocked on it. `EB_TENANCY_MODE=single` remains the default and reproduces Phase 2. eventbridge/owner_index.py — the one deliberately cross-tenant table `correlationid -> userkey`, UNIQUE on `correlationid` **alone**. That is the part to get right: `(userkey, correlationid)` is the correct primary key inside a tenant's store and exactly the wrong constraint here, because two tenants minting one id produce two distinct tuples, never conflict, and the collision passes through silently — which is the failure the guarantee exists to prevent. It has to be cross-tenant: the lookup that decides which tenant owns a correlation must happen before a per-user store can be chosen. It is also what lets `sessionuuid = uuid5(NAMESPACE, correlationid)` stay UNSALTED (§2.6) — salting it would break every existing session for a collision this constraint removes. `owner_of` returns `(userkey, known)` because a bare `None` conflates "the shared tier owns this" with "never seen", and §6.2's 404 rule needs them distinguished. eventbridge/store_registry.py — per-user stores, LRU-bounded Separate FILES, not one database with a `userkey` column. `store.py` has 28 methods and every one touches tenant data; a `WHERE userkey = ?` remembered 28 times is one that will eventually be forgotten, with a user reading someone else's conversation as the symptom. Separate files make the mistake structurally unavailable — the connection IS the tenant. Two connections per tenant in WAL mode (main + -wal + -shm) bounds this near 150-200 tenants under a 1024 soft RLIMIT_NOFILE, hence `EB_MAX_OPEN_STORES=64`. A store with live SSE subscribers is PINNED: evicting one makes an open page stop updating with no error anywhere. If everything is pinned the cache exceeds its ceiling rather than breaking a viewer — over-budget degrades and is visible on /healthz; a dead stream does not recover. Minter now consults the index (§2.6) One indexed SELECT per mint replaces the startup seeding loop. With N per-user stores the old loop meant N SQLite opens before the socket binds — 100 tenants x 10,000 correlations for a guarantee one SELECT gives directly. The claim happens inside the retry loop, so a candidate lost to a concurrent worker retries instead of both callers believing they own it. `mint_for()` passes `userkey` only to minters that accept it, by signature inspection rather than by catching TypeError around the call — a working `mint()` can raise TypeError from its own body, and swallowing that would silently drop the ownership claim. Routing Responses are filed by the event's OWN signed `ce_userkey`. One with no userkey in multi mode lands in `shared/` and increments `unattributed` (now on /healthz): never guessed into a tenant's store, never dropped. `/continue`, the reads, the group routes and the deadline sweeper all route through the index — the sweeper across every tenant, because a stalled batch belonging to an untouched tenant is exactly what deadlines are for and also the least likely to be in the LRU. This retires the submitter-derived owner lookup from the previous commit, and with it the limitation that it could not tell a GitHub `alice` from a static `alice`. Single-tenant mode keeps its files exactly where Phase 2 left them — directly in the bridge root, not under `shared/` — so an upgraded deployment's sessions and transcripts keep showing up in the UI instead of appearing to vanish. Two bugs found and fixed while building this: * **Eviction could return a closed store.** With every older entry pinned, the LRU took the store it had just opened and handed the caller a closed one — surfacing later as `ProgrammingError: Cannot operate on a closed database` from somewhere that looks unrelated to caching. Fixed with `protect=`, with a regression test. * **`except Exception` hid a signature mismatch.** Adding `userkey=` to `publish_group_event` made every group event stop publishing, and the broad handler around it turned that into a batch that silently never completed. TypeError now re-raises at both call sites. `Store.close()` added (it had none), and it wakes subscribers first so a viewer blocked on `Event.wait()` notices shutdown instead of hanging to its own timeout. Tests: 49 new (785 total), ruff clean. Verified both modes start and that the on-disk layout is upgrade-safe. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): evict stores by dropping the reference, not by closing them Found reviewing T5. `StoreRegistry` eviction called `Store.close()`, which created a use-after-close race on every multi-call read path: a handler resolves a store, and while it is still making calls a *concurrent request for a different tenant* pushes the cache over `EB_MAX_OPEN_STORES` and closes it underneath. The next call fails with sqlite3.ProgrammingError: Cannot operate on a closed database from a line that looks nothing to do with caching. `get_html` alone makes five calls on one store, so the window is wide rather than theoretical, and reproducing it needs only `max_open=1` and two tenants. Eviction now drops the registry's reference and lets CPython close the connections when the last holder goes away — exactly the lifetime that is safe. File descriptors are reclaimed slightly later than an explicit close; that is the right trade, because the ceiling is a soft budget while a closed connection under an in-flight request is a 500. `_sse_generator` additionally now subscribes BEFORE the replay read. Pinning is what keeps a subscribed store in the cache so later readers get the same object and see the subscriber's notifications, and the old order left the store unpinned between `_store_of()` and `subscribe()`. Reordering is harmless: the Event only ever means "there may be something new", every read is `since_seq=sent`, and an already-set Event costs one extra query. Worth noting what the first fix attempt got wrong, since it is the more interesting mistake: pinning only the SSE path would have closed the case that was easiest to see while leaving every ordinary read exposed. The bug was in the eviction contract, not in one caller. Two regression tests, one asserting an evicted-but-held store stays usable for both reads and subscribe, one pinning the subscribe-before-replay order. 787 passing. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): teach GroupMirror about tenancy, and stop hiding index failures Three bugs from an adversarial review of T5. All three were confirmed by driving the real code, and all three are multi-tenant-only — the default `single` path is unaffected. **1. Restart-orphaned groups were never settled (`_settle`).** It called `maybe_complete(gid)` with no `userkey`, so it looked in the SHARED store for a group that lives in a tenant's store, found nothing, and returned False. Every group a restart left unfinished stayed unfinished forever, and the failure mode is silence — which §21.6 calls the worst one. The mirror now resolves the owner per group from the ownership index. **2. Replayed member events landed in the shared store.** `on_member_event` reads `ce_userkey` off the event, and any response published *before* Phase 3 has none — so member rows went to `shared/` while the group row sat in the tenant's store, leaving the group's counts permanently wrong and `_decorate_member` reading from a third place. Pre-existing Kafka history is exactly what this mirror exists to replay, so that is the normal upgrade path, not an edge case. The mirror now back-fills a missing `userkey` from the index. It only ever FILLS: an event carrying its own `userkey` keeps it, because that attribute is signed (§2.6) and the index is not authoritative over a signed assertion. Both of these had one root cause — the mirror knew about per-user topics but not about which tenant a replayed group belonged to. **3. A broken index surfaced as "correlation ID space exhausted".** `mint()` caught `Exception` around `claim` to treat a lost race as a retry, so a genuinely broken index (disk full, database locked, schema missing) was retried 2000 times and then reported a message that sends the reader to the word lists rather than the database. Now catches `Collision` specifically; everything else belongs to the caller. Also closes a test-strength gap the review found: deleting the `_has_subscribers` pinning check entirely still passed every `test_store_registry.py` test. The existing tests asserted object identity and counters, not the notification path that is the actual reason for the rule — now that eviction only drops a reference, a held store keeps working, so what actually breaks is that the *writer* gets a new Store whose subscriber map is empty and the viewer is never notified. The new test asserts that end to end; removing the pinning check now fails 4 tests instead of 0. Tests: 8 new (795 total), ruff clean. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * docs(eventing): record T5 as implemented, and what it does not yet give you The status lines said steps 5-12 were design only; T5 (per-user stores and the global ownership index) is now in the tree, so they were wrong in the direction that matters — a reader would either re-implement it or, worse, read "per-user stores" as "isolation". The distinction the docs now draw explicitly: each tenant's sessions, responses and transcripts DO live in their own SQLite files, and a read routes to the owning tenant's store — but nothing yet checks that the *caller* is that owner, because owner-scoped reads (T7) and transcript auth (T8) are not implemented. So `multi` mode's reads, `/continue` and `PUT /transcript` remain as open as Phase 2's. Also adds `EB_MAX_OPEN_STORES` to the component README's Phase 3 table with the descriptor arithmetic that explains the default, and to `test_manifests.py`'s classification list so T22's "every Phase 3 variable is on one side of the Secret-vs-ConfigMap rule" check keeps covering it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): address the five must-fixes from the #883 review Every finding reproduced against 16158e5 before fixing, and every fix has a regression test in `tests/test_review_fixes.py` that fails without it (verified by mutation, one fix at a time). They are filed together because they share a cause worth naming: all five were invisible to 192 test functions, because the fixtures assembled the collaborators by hand with the arguments the production call sites fail to supply. **1. `single`-mode regression: the ownership index was never seeded.** The serious one, and against the exact guarantee this phase claims. Dropping Phase 2's `Minter` seeding loop left nothing to populate its replacement, so on the first start after an upgrade `owners.sqlite` is empty while `sessions.sqlite` is full and `exists()` reports live ids as free. With ~1,000 existing correlations a mint has roughly 1-in-25,000 odds of reissuing one, and then `upsert_session` overwrites a session and the new prompt appends to somebody's existing conversation. Phase 2's odds were zero. `OwnerIndex.seed_from()` plus a one-time backfill from every store at startup. §2.6 is right that per-start seeding is the wrong trade; it accidentally argued away the one-time backfill too, and §6.1a now records the distinction. **2. `create_group` did not pass the userkey.** `submit_members` got it and `groups.create()` did not, so in `multi` the group row and the groupid's owner claim landed in the shared store while every member row landed in the tenant's: 0 members forever, `maybe_complete` never fires, the batch never completes and never notifies. A retried POST with an Idempotency-Key read the tenant store `create` never wrote and returned `members: []` — worse than a second batch, because it looks like success. **3. `close_group`/`cancel_group` were unauthenticated cross-tenant mutations.** Neither called `_caller`; both resolved any tenant's group from the global index and acted on it. T7 covers *reads* staying as open as Phase 2's — Phase 2 had no tenants to cross and no cross-tenant mutation anywhere, so this was a capability being introduced, not an openness preserved. Now gated by `_authorize_mutation`, answering `404` for another tenant's group per §6.2's enumeration rule. Single-tenant mode is untouched. **4. `agent` was outside `SIGNED_ATTRS`** while `ce_agent` selects the AgentSpec that supplies `--permission-mode` and the tool allowlists — §5.1's "sandbox". A forged value widened the sandbox and the signature still verified, making a signed deployment worse than an unsigned one because an operator believes it is attested. Added in the same change as `userkey`/`depth`, per §8.3's one-canonicalisation-break rule. **5. `userkey` reached a filesystem path join unvalidated**, from an inbound Kafka header, and `Store.__init__` calls `mkdir(parents=True)` — so `..` created and wrote to a directory inside another tenant's store, or outside the tree. `tenancy.is_valid_userkey()` is the gate, exported from the module that is "the ONLY place an identity becomes a name" so a consumer can check the shape the producer guarantees. An invalid key is treated as a missing one — `shared/` plus `unattributed` — so a forgery is counted and visible rather than acted on. Note it is *truthy*, so the pre-existing `not userkey` check let it past. Also from the review: - **A known-but-unowned correlation 500'd on `/continue`.** `owner=None, unresolved=None` is falsy, so the handler proceeded, wrote a prompt row, then `TopicSet.requests(None)` raised — a traceback with an orphan row already committed, so the conversation showed a turn that was never submitted. Reading and publishing have different requirements, so `_publishable_owner` is now separate from `_owner_userkey`: shared-tier data stays readable, and a turn for it is refused before anything is written. - `claim()` uses `ON CONFLICT DO NOTHING` + a follow-up read, so `Collision` is the only way a conflict presents whether or not two writers race the in-process lock — the bare `INSERT` would have raised `IntegrityError`, which `Minter.mint` deliberately does not catch. - `created_utc` at millisecond precision (matching `ce.now_iso()`) with a `rowid` tiebreaker; at second granularity a 100-member batch had arbitrary order among its rows. - A bad `agent` name is a `400` on the call that made the mistake, not a `202` followed by an async `phase=error` the submitter may never read. - `max_turns: 0` is no longer silently rewritten to the spec's value while the spec path rejects zero. - `EB_TOPIC_PREFIX` validated (it is the one topic-name component `_slug` never sees, and `.`/`_` there reproduce the JMX metric collision) and `EB_MAX_OPEN_STORES` refuses readably instead of raising a bare `ValueError`. - Dropped a flaky assertion: `"gh" not in` 26 characters of base32 fails ~2.4% of the time by construction, and the `"mrsabath" not in` line above it is the real property. - `_canonicalise`'s docstring no longer claims static identities are byte-exact when the `@` branch catches address-shaped ones first. Design doc, where the code had disproved it (§9's "the code is the authority" rule): §6.1's layout corrected to put single-tenant stores in the bridge root; "Closing is safe" replaced with what commit 49aed05 established, including that the first fix attempt only pinned the SSE path and left the class open; new §6.1a documents `owner_index.py`; §2.6's "one new extension attribute" corrected to four, with which three are signed and why; §3.1 records the single-mode `events`/`dead` names. The status header now warns that `multi` does not work end to end until T6 — no response is consumed, which is more useful than "not yet isolated". Tests: 45 new (840 total), ruff clean. Still outstanding from the review and deliberately not in this commit: the T22 subset check, `RequestsMirror`/`NtfyPublisher` `stores=`, the handler-to-topic integration test, the `forget()` tombstone decision, and the four documents. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): the four remaining #883 suggestions, and tombstone deletion Round two of the review, covering what was deferred from 6ebdd80. **T22 now checks the direction that enforces the rule.** `CONFIGMAP_VARS ⊆ found` catches a stale entry and nothing else; the direction that catches a *new* variable classified as neither is `found ⊆ CONFIGMAP_VARS ∪ SECRET_PATH_VARS`, which is the case the test exists for. Scoped to Phase 3's variables — the 36 that pre-date §8.3 are listed explicitly, with `test_the_baseline_list_is_still_accurate` keeping that list honest rather than letting it silently widen the exemption. Verified by deliberate failure, as asked: injecting an unclassified `e("EB_UNCLASSIFIED_X")` read into `config.py` turns the test red with the variable named. Also replaced the hardcoded ten-space indentation in the inlined-secret check with `\s*`, matching the precedent elsewhere in the same file — the old pattern would miss the same variable inlined in an initContainer, a sidecar or a deeper patch. **`RequestsMirror` and `NtfyPublisher` now take `stores=`.** Both knew the per-tenant topic names and still wrote to the shared store, which does look like the sweep in 77807b6 not reaching them. The mirror back-filled every prompt into `shared/`, so a correlation whose request the bridge did not originate showed a BLANK PROMPT on its owner's page — the exact gap the mirror exists to close. ntfy's `_last_prompt` and `_last_assistant_text` read only the shared store, so a tenant's notification arrived without the prompt or the reply, which §4.1 says is the content that lets a phone that cannot reach EventBridge still see the result. Its group-existence check is routed too. **A handler→topic integration test**, which the review argued was a better investment than more unit tests on the pieces. It is: faking at `KafkaProducer` rather than at `Producer` means the real handler, the real `Producer` and the real `TopicSet` all run, and the assertion is *which topic the bytes went to*. Confirmed by reverting each fix that it catches both defects it was written for — the `create_group` missing `userkey` and the `/continue` 500 on an unowned correlation. That test immediately earned itself twice over: the mutation run surfaced that `_publish_started`/`maybe_complete` swallow `ValueError` from `TopicSet` into a log line, so a caller forgetting a `userkey` produced a group that silently never announced itself — the same shape as the `TypeError` already guarded there. Both guards now cover both. **Deletion tombstones rather than freeing the id.** This is the one I said needed thought rather than a quick answer, and the answer is that reuse is unsafe. Verified: `forget()` used to `DELETE` the row, `exists()` is the `Minter`'s only uniqueness check, and `sessionuuid = uuid5(NAMESPACE, correlationid)` is unsalted by design — so a reissued `correlationid` derives *the same session uuid*, and a `claude` transcript left on a runner's volume becomes resumable by the new correlation. That is one user's conversation continuing inside somebody else's agent. So `forget()`/`forget_tenant()` keep the row, set `deleted_utc` and clear `userkey` — the id is reserved forever, and the tombstone discloses nothing about whose it was. The alternative is guaranteeing every transcript keyed on that uuid is purged everywhere, including volumes this process does not own, which is not a guarantee EventBridge can make. Cost is one short row per deleted correlation. Migrated with an `ALTER TABLE`, the same idiom `store.py` uses for `prompts.submitter`, and a test covers an index created before the column existed. The reviewer was also right that the docstring contradicted the code: it claimed "freeing the id for reuse is deliberate" about a `DELETE` nobody had reasoned about. Design doc: §6.1a records the tombstone decision and why, since the index is what enforces it. Tests: 855 passing (+13), ruff clean. Still outstanding: the four phase documents (`IMPLEMENTATION_REPORT3.md`, `README_PHASE3.md`, and the two owed Phase 2 documents). Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * docs(eventing): IMPLEMENTATION_REPORT3.md — what Phase 3 built, measured Asked for in the #883 review, and the argument for writing it now rather than later was the most persuasive one there: the findings are attributable to specific commits today and will not be in a month. 361 lines, following `IMPLEMENTATION_REPORT1.md`'s contract — measured figures rather than estimates, failures with root causes, and an explicit account of what is NOT verified. Phase 0 and Phase 1 each ship three documents; Phase 2 shipped none, and the gap shows in that there is nowhere to put the §6.1 split-consumer trap except inside a design document. This starts closing that for Phase 3. **Measured rather than asserted.** §6.1 estimates the per-tenant descriptor cost and derives a ceiling; measured it is 6.0 descriptors per open store (10 tenants, 60 fds), which puts ~680 tenants under this host's 4096 soft limit and ~170 under the 1024 the design assumed — so squarely inside its "150-200" once the limit is held constant. The ownership index costs 27.4 µs per `claim`, 1.7 µs per `exists`, and 270 ms to seed 10,000 ids. That last figure is the one that matters: it settles §2.6's objection to seeding, because the *one-time* backfill it appeared to rule out costs 270 ms once, while the *per-start* seeding it correctly rejects is what the index exists to avoid. **Seven deltas from the design, recorded as decisions rather than drift** — the single-tenant store location, eviction by reference-drop, tombstoned deletion, the single-mode `events`/`dead` names, `agent` joining `SIGNED_ATTRS`, the exported userkey validator, and prefix validation. **The findings, ordered by what they cost to find.** Three are worth the document on their own: - The `single`-mode regression shipped behind a headline claiming it could not happen, with the reproduction (50 of 50 live ids reported free), the 1-in-25,000 arithmetic over a verified 50x50x10,000 id space, and the three reasons it is instructive: it was in the default configuration, the suite could not see it because no fixture simulated an upgrade, and §2.6's wording contributed. - The eviction contract, where the lesson is the *first* fix: pinning only the SSE path addressed the case that was easy to picture and left the class open. - `except Exception` converting three separate programming errors into log lines, on three separate occasions, which is why it is filed as one finding with the pattern extracted. Also records the suite's structural blind spot as a finding in its own right — a fake placed at the boundary under test cannot test that boundary — since that is what let two defects survive 200-odd new test functions. **§5 is the longest section, deliberately.** `multi` does not work end to end until T6 (no response is consumed); no cluster, no two-tenant deployment, no Kafka; reads not owner-scoped; Tier A is not isolation; the per-tenant throughput ceiling §10 asks for is still unmeasured; the `claude --help` flag assertion does not run in CI; `seed_from` is capped at 10,000 per store; and `is_valid_userkey` is shape validation, not authorisation. Every figure in the report was measured on this branch, every cited commit verified to be on it, and every cited design section verified to exist — §21.6 turned out to be `DESIGN_PHASE1.md`'s and is now qualified as such. Two numbers in my own PR description were wrong and are corrected here: the test delta is +232 unique functions (525 to 757), not the 139 or 217 claimed earlier. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * fix(eventing): two orphan-write paths, and three design/code gaps (#883 round 2) Both must-fixes reproduced against c25a0c7 before fixing, and both are my own round-1 fix introducing a new bug class: `6ebdd80` added validation of the request's `agent` name at the HTTP boundary — right place, right reason — and placed it AFTER the writes. **create_group** had already committed the group row and published `group.started` when the 400 returned, and because the row is keyed on the Idempotency-Key the state was permanent: every retry read back `created: false, members: []`, which looks like success. That is verbatim the failure the `userkey=` comment added in the same round describes, reached by a route that fix did not cover. POST /v0/groups {"prompts":["a","b"],"agent":"../../etc"} Idempotency-Key: retry-me -> 400, topics: ['kev1-st-alice-c8ff0431-responses'], groups: ['wild-weasel-1995'] retry, same key, valid name -> 200 {created: false, members: []} **start_agent** had written the session row, the prompt row and claimed the correlation id — a transcript showing a turn that was never submitted, plus an id nothing ever un-claims, since tombstoning is for deletion rather than abandonment. Both resolutions now happen before the first write. `_agent_for` needs only `caller` and `body`, so there was never anything to gain by deferring it. After the fix both paths publish nothing, write nothing, and the group retry launches the batch properly. The reviewer's observation about the tests is the more useful half: the assertion that would have caught both — `get_prompts(corr) == []`, *a refusal writes nothing* — already existed 25 lines earlier in the same file, on a different route. The new tests assert it on both paths, with the Idempotency-Key retry for the group case, and were verified by reverting each ordering (7 failures). ### Three design/code gaps §3 of the report had missed **The backfill was per-start, not one-time.** §6.1a says "seeded once" and the report said "once, on the first start after an upgrade"; the loop re-opened every tenant store and re-read `all_correlations` on every boot — exactly the N-store startup seeding §2.6 rejects, reintroduced by the fix for the missing seed. `INSERT OR IGNORE` kept it correct; what repeated was the cost §2.6's argument is entirely about. A `seed_state(scope, done_utc)` table now records each scope, and `needs_seeding` is checked before the store is opened. Measured at 100 tenants x 200 correlations: before: start 1 702 ms / 101 opens start 2 210 ms / 101 opens after: start 1 702 ms / 101 opens start 2 0.8 ms / 1 open Sound because nothing can add an unindexed id to a seeded store: every mint claims as it mints. **`AgentSpec.limits` was parsed and enforced nowhere, silently.** `Skill` states plainly that it carries `sha256` for T19; `Limits` made no such disclaimer, so `timeout_s = 900` loaded clean and did nothing. Now says so, with which field belongs to which later task. Its three conversions also escaped the module's own `SpecError` contract — `timeout_s = "900s"` raised a bare `ValueError` past `run_agent`'s handler instead of becoming the `phase=error` event §5.1 promises. **§8.3 disagreed with §2.6 and the code.** It still said the signed set "changes twice" and named two attributes, while §2.6, `signing.py` and `ce.py` all said three — and `signing.py` cited §8.3 as the authority for a rule §8.3 stated over the wrong count. Two smaller ones from the same pass: a tombstoned correlation could be re-claimed by the NULL owner (`row[0] == userkey` reads as "already mine" when both are NULL), contradicting what `forget` guarantees — unreachable from `Minter.mint`, which checks `exists()` first, but reachable as soon as a correlationid arrives from outside, which §7's triggers will bring. And `registry.parse` accepted `agent` unchecked, the only field it did not validate, in a function whose docstring promises every inconsistency is a refusal; a typo there meant an asynchronous `phase=error` on every request from that user rather than a startup refusal. Verified: the registry now refuses with the bad value named. ### Documentation - The component README's Phase 3 block no longer orphans the "Two of these are load-bearing" sentence from its payoff 25 lines later — the block moved below it under its own `#### Phase 3` heading, and the note count is now right (three, not two). Added the T6 "does not work end to end" warning here too. - `IMPLEMENTATION_REPORT3.md` §1 now counts from a named point (`3278d75`) instead of "currently", since a report cannot be right about the diff of the commit containing it; both figures are given. New §4.7 and §4.8 record this round's findings, including that fixing one orphan-write instance did not teach me to look for the class — the same lesson §4.2 records about the eviction contract, learned a second time. Tests: 860 passing (+5), ruff clean, both multi-mode startup paths smoke-tested. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * test(eventing): Phase 3 — regression tests for the three untested fixes (#883) From the #883 approval: reverting `agentspec`'s `try/except`, `claim`'s tombstone check or `registry`'s `validate_name` left all 858 tests passing, so three of the six fixes in `258da7a` were invisible to the suite. The standard this PR has applied since `6ebdd80` is "every fix has a regression test, verified by reverting its fix", and it had been applied to half that commit — and, as the review put it, the three without one are the three whose absence is invisible. Fourteen tests, each verified by reverting its fix independently: | Fix | Failures when reverted | |---|---| | `[limits]` conversions wrapped in `SpecError` | 4 | | tombstone never re-claimable | 1 | | `registry.parse` validates `agent` | 5 | The tombstone test covers the case that was actually wrong — `claim(corr, None)` against a tombstoned row, where `row[0] == userkey` is `None == None` and read as "already mine" — and also asserts a *tenant* re-claim now fails with "tombstoned" rather than "already owned by (single-tenant)", so the message points at the real reason. The registry tests cover the shapes `validate_name` rejects and, separately, that a valid name loads and an absent one stays empty so `ER_AGENT_NAME` still applies. A gate that rejects what it should allow is the other way to get this wrong. **Also closes the range-check gap the review flagged as on-the-record.** `timeout_s = -5` and `max_events = 0` loaded clean: timeout_s = -5.0 | max_events = 0 Defensible while `Limits` is documented inert, but a negative deadline is nonsensical input whether or not anything reads it yet, the module's own standard is that a policy which cannot be understood fails to load, and the validation was already right there. Rejecting now also means whoever wires up the `Popen` wrapper inherits a value they can trust instead of re-validating. `0` keeps its documented meaning for `timeout_s` and `max_output_bytes` ("no deadline" / "unbounded"), with a test pinning that the range check does not reject the sentinels; `max_events = 0` is refused because a run that may emit no events cannot report its own result. Tests: 875 passing (+14), ruff clean. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * test(eventing): extend the §8.3 rule to #884's seed paths, and cover kind-signed Follow-up to rebasing onto `main` now that #884 and #890 have merged. Both predicted conflicts were in `agentdocs/README.md` and resolved as planned — `main`'s Phase 2 row (which gained §8 from #890) plus this branch's Phase 3 row. `test_manifests.py` auto-merged, keeping #884's `allowed_secrets` fix beside this branch's Phase 3 classification. That left two things the merge did not resolve by itself. **Extending `SECRET_PATH_VARS` to #884's seed paths needed the rule split, not widened.** #884's review predicted this tuple would be extended to `ER_/EB_SIGNING_KEY_PATH` and checked that doing so stays green. It does for the ConfigMap half and would NOT for the literal-value half, because those two variables hold a **mount point, not key material** — and #884 deliberately sets them as container `env` entries so the path sits beside the volume that supplies the seed, which is what `configmap.yaml` asks for. So the tuple is now two: - `SECRET_VALUE_VARS` — the three whose *value* is key material. No literal value anywhere, which is the original assertion. - `SEED_PATH_VARS` — #884's two. A literal value is correct; a ConfigMap key is not. Both still obey "never from `config.toml`", so that check covers all five now where it covered three. Treating a path naming a secret as if it were the secret would have forced #884 to either revert its own correct fix or carve itself out of the rule — the distinction is the thing worth encoding. **`kind-signed` was not in `OVERLAYS`.** #884 added the overlay and every assertion in this file silently skipped it — including the seed-path rule the PR exists to respect, the `replicas` invariants, and the no-credential checks. Added, which is also what makes the new assertion meaningful: with `kind-signed` absent, injecting a seed path back into `eventing-config` passed; with it present, that regression fails as it should. Verified both ways. Tests: 899 passing (875 here + #884's 24 picked up in the rebase), ruff clean, all four overlays build. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> --------- Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
Phase / scope
DESIGN_PHASE2.md), the deployment halfuserkey.agentdocs/and this one changes none of it. #883 (Phase 3 tenancy) — no overlap in files; it does addEB_/ER_variables, so whichever merges second re-runstest_manifests.py's Secret-vs-ConfigMap classification.Draft — about half the plan. The key material is generated, mountable and applied at deploy time; the preflight check, e2e assertions, the forged-response demo stage and the
README_PHASE1rewrite are still to come (list at the bottom). Opening early because the Secret-handling and manifest decisions are where review is cheapest, and because one of them deviates from a documented convention on purpose.The problem
#879 wired signing in both directions, but none of it could be switched on in a cluster. The ConfigMap shipped
ER_/EB_VERIFY_KEYSET_PATHand*_SIGNING_KIDas empty strings andconfigmap.yaml:54said seed paths "belong in the Deployment beside the volume" — except no Deployment had one. The only volume in either wasdata: emptyDir, andgrep -rn "secretName\|configMap:" k8s/returned nothing. So the feature worked with local file paths on a laptop and was unreachable in Kind or OpenShift.That blocks the demo track: S6, S8 and S9 (#2596/#2598/#2599) are each defined as "reproducible on a local Kind cluster", and S9 is the forged-event teaching moment.
There was also no way to make a key.
shared/signing.pyhaspublic_key(seed)but nothing anywhere generated one — enabling signing meant hand-rollingos.urandom(32).hex()and deriving public keys in a REPL.What's here
scripts/gen_signing_keys.py— seeds at mode 0600 under a 0700 directory, and anagents.jsonwhose public keys are derived from those seeds rather than typed, so the keyset cannot disagree with the keys it authorizes. Never prints a seed on either stream; only kids and public keys, because a pasted terminal transcript is the most likely way a demo key escapes. Refuses to overwrite an existing seed without--force, since that key may already be deployed and in an approved keyset.Both Deployments mount a seed Secret at
/etc/ce-signingand the keyset ConfigMap at/etc/ce-keyset, bothoptional: true— without that, every signing-less overlay would block inContainerCreatingwaiting for a Secret that will never exist.k8s/base/keyset-configmap.yamlships empty. "There is never a key in this file" is checkable; "only public keys in this file" is not, and a 32-byte seed is indistinguishable from a public key by length.k8s/overlays/kind-signed/flips enforcement on. Separate overlay rather than a patch onkind, because S3/S5 are explicitly unsigned milestones and the before/after contrast is the demo.k8s_deploy.py --signing-keys DIRapplies one seed Secret per service plus the keyset ConfigMap, via stdin so the seed never reachespsthrough argv — modelled onensure_ntfy_secret, including the no-op-when-absent behaviour so a redeploy can't silently switch enforcement off and strand a signed topic. It validates first, because every one of these produces a cluster that refuses all traffic with no local symptom: the directory and each named seed must exist,agents.jsonmust parse and approve at least one kid, and the kid each service signs under must be in that keyset.One deliberate deviation, and one bug caught before it shipped
Per-service Secrets, not the documented single one. Both Deployments initially mounted the same
event-signing-key, which would have given the two services one shared seed. That collapses the asymmetry the design rests on — EventBridge would hold the key it verifies runner responses with and could forge them, which is the exact property that made HMAC unacceptable (DESIGN_PHASE2.md§4.3). Noweventbridge-signing-keyandeventrunner-signing-key, with a test asserting they never match.This deviates from a runbook, not a design — a correction from review worth stating
plainly, because I had written it up as a bigger decision than it is.
DESIGN_PHASE2.md§4.4 and §5 say nothing about Secret names or count, and
DESIGN_PHASE1.md§11's"Ed25519 keys in a read-only Secret" is satisfied by two as easily as by one. So it costs
the README rewrite already owed, not a design amendment.
It deviates from
event-signing-keyas documented atREADME_PHASE1.md:704-722. That section needs rewriting regardless — it tells the operator to write a seed to/tmp/seed.hexandrmit, contradicting the stdin-only rule stated 230 lines earlier at:475, and it says "have the publisher sign with the same seed", describing the symmetric setup §4.3 explicitly rejected. The rewrite is in the remaining work; flagging it now so the name change isn't a surprise.Logging asymmetry is intentional. Seeds are logged as a length only — not even a prefix, unlike the API token, because 4 hex chars of a 32-byte seed is 16 bits of a private key and there's nothing to gain from identifying it. Kids are logged in full, because "the operator deployed yesterday's key set" is otherwise invisible until it surfaces as an unexplained rejection.
Also fixed a pre-existing leak:
--purgelistedeventbridge-ntfyin neither thedoomedpreview nor the delete tuple, so a "purged" namespace kept an ntfy topic — a capability that lets anyone who learns it both read and publish your notifications.Testing
568 passed, 5 skipped (5 cluster-gated). 17 new tests.
ruff checkandruff format --checkclean.The three manifest tests this could have broken all still pass, and they're why the design looks the way it does:
test_no_credential_is_ever_rendered_into_a_manifest(so the Secret is applied by the script, never declared — nosecretGenerator),test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic(so enforcement stays out of the shared overlays), andtest_no_manifest_sets_run_as_user(so nodefaultMode: 0400, which would be root-only given OpenShift assigns the UID).Verified by rendering rather than assumed:
datais still volume index 0 in all three existing overlays, so the demo overlay'sop: replaceon/spec/template/spec/volumes/0still targets the PVC swap it means to.kind: Secret.kindstays fully off;kind-signedrenders enforcement plus both key paths, and those values map onto the realCfgfields.verify_with_keysetafter ato_kafka_binary/from_kafka_binaryround trip.Not verifiable in my sandbox, and not claimed:
kubectl apply --dry-run=clientneeds a reachable cluster to fetch the OpenAPI schema and fails even with--validate=false, and Kind can't run here (podman has no VM; creating one fails on the proxy's download cap). So schema validity of the new volume stanzas and the actual mount behaviour are unproven — covered bytest_every_object_is_accepted_by_the_api_serverwhen run against a real cluster.Still to do on this branch
k8s_preflight.pycheck 14 — Secret present, keyset parses, every kid the ConfigMap names is in itk8s_e2e_test.py— assert the terminal response carriesce_signatureand the startup banner says signing is ONk8s_demo_flow.py --forge— publish an unsigned event to the responses topic and assert it lands asphase="error"with the payload retained underdata["rejected"]. Mechanism already validated against a live broker:kafka-console-producer.sh --property parse.headers=trueyields a CloudEvent the consumer parses and the verifier rejects (binary mode matters — a structured-mode body givesattrs == {})README_PHASE1.md§"Signed events"Question for review
Is
kind-signedthe right shape? The alternative is a--signedflag on the existingkindoverlay. I went with a separate overlay so the unsigned path keeps working for S3/S5 and so thetestoverlay's no-credential assertion keeps meaning something — but the name is about to get baked into three more files (preflight, e2e, demo flow), so it's cheap to change now and annoying later.Separately: S6 (#2596) specifies ES256 + a SPIFFE-derived
ce-source+ rotation, and this is Ed25519 + a static keyset. The eventing design docs never commit to ES256 — only the issue does, and §4.3 argues Ed25519 on merits with SPIRE as a later swap that leaves verification logic unchanged. Worth an explicit decision rather than a surprise at demo time; I'll comment on #2596.Assisted-By: Claude Code