Skip to content

feat(eventing): Phase 2 — make event signing deployable on a cluster - #884

Merged
mrsabath merged 3 commits into
mainfrom
feat/eventing-signing-deployable
Oct 5, 2026
Merged

mrsabath merged 3 commits into
mainfrom
feat/eventing-signing-deployable

Conversation

@mrsabath

@mrsabath mrsabath commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Phase / scope

Phase Phase 2 — identity on the event path (DESIGN_PHASE2.md), the deployment half
What this changes Makes Phase 2's signing switchable on a cluster. #879 wired it in both directions; none of it could be turned on in Kubernetes, because no Deployment mounted a seed and the ConfigMap shipped the paths empty.
Phase 2 sections it implements §4.3 (Ed25519 over HMAC, and why the two services must not share a seed), §4.4 (the two-flag rollout), §5 (the Secret-vs-ConfigMap rule)
Phases it does not touch Phase 0/1's wire contract and KEDA model are unchanged. Nothing here is Phase 3: no tenancy, no per-user topics, no userkey.
Relationship to the other drafts #890 (Phase 2 docs) — no overlap, that PR changes only agentdocs/ and this one changes none of it. #883 (Phase 3 tenancy) — no overlap in files; it does add EB_/ER_ variables, so whichever merges second re-runs test_manifests.py's Secret-vs-ConfigMap classification.

Draft — about half the plan. The key material is generated, mountable and applied at deploy time; the preflight check, e2e assertions, the forged-response demo stage and the README_PHASE1 rewrite are still to come (list at the bottom). Opening early because the Secret-handling and manifest decisions are where review is cheapest, and because one of them deviates from a documented convention on purpose.

The problem

#879 wired signing in both directions, but none of it could be switched on in a cluster. The ConfigMap shipped ER_/EB_VERIFY_KEYSET_PATH and *_SIGNING_KID as empty strings and configmap.yaml:54 said seed paths "belong in the Deployment beside the volume" — except no Deployment had one. The only volume in either was data: emptyDir, and grep -rn "secretName\|configMap:" k8s/ returned nothing. So the feature worked with local file paths on a laptop and was unreachable in Kind or OpenShift.

That blocks the demo track: S6, S8 and S9 (#2596/#2598/#2599) are each defined as "reproducible on a local Kind cluster", and S9 is the forged-event teaching moment.

There was also no way to make a key. shared/signing.py has public_key(seed) but nothing anywhere generated one — enabling signing meant hand-rolling os.urandom(32).hex() and deriving public keys in a REPL.

What's here

scripts/gen_signing_keys.py — seeds at mode 0600 under a 0700 directory, and an agents.json whose public keys are derived from those seeds rather than typed, so the keyset cannot disagree with the keys it authorizes. Never prints a seed on either stream; only kids and public keys, because a pasted terminal transcript is the most likely way a demo key escapes. Refuses to overwrite an existing seed without --force, since that key may already be deployed and in an approved keyset.

Both Deployments mount a seed Secret at /etc/ce-signing and the keyset ConfigMap at /etc/ce-keyset, both optional: true — without that, every signing-less overlay would block in ContainerCreating waiting for a Secret that will never exist.

k8s/base/keyset-configmap.yaml ships empty. "There is never a key in this file" is checkable; "only public keys in this file" is not, and a 32-byte seed is indistinguishable from a public key by length.

k8s/overlays/kind-signed/ flips enforcement on. Separate overlay rather than a patch on kind, because S3/S5 are explicitly unsigned milestones and the before/after contrast is the demo.

k8s_deploy.py --signing-keys DIR applies one seed Secret per service plus the keyset ConfigMap, via stdin so the seed never reaches ps through argv — modelled on ensure_ntfy_secret, including the no-op-when-absent behaviour so a redeploy can't silently switch enforcement off and strand a signed topic. It validates first, because every one of these produces a cluster that refuses all traffic with no local symptom: the directory and each named seed must exist, agents.json must parse and approve at least one kid, and the kid each service signs under must be in that keyset.

One deliberate deviation, and one bug caught before it shipped

Per-service Secrets, not the documented single one. Both Deployments initially mounted the same event-signing-key, which would have given the two services one shared seed. That collapses the asymmetry the design rests on — EventBridge would hold the key it verifies runner responses with and could forge them, which is the exact property that made HMAC unacceptable (DESIGN_PHASE2.md §4.3). Now eventbridge-signing-key and eventrunner-signing-key, with a test asserting they never match.

This deviates from a runbook, not a design — a correction from review worth stating
plainly, because I had written it up as a bigger decision than it is. DESIGN_PHASE2.md
§4.4 and §5 say nothing about Secret names or count, and DESIGN_PHASE1.md §11's
"Ed25519 keys in a read-only Secret" is satisfied by two as easily as by one. So it costs
the README rewrite already owed, not a design amendment.

It deviates from event-signing-key as documented at README_PHASE1.md:704-722. That section needs rewriting regardless — it tells the operator to write a seed to /tmp/seed.hex and rm it, contradicting the stdin-only rule stated 230 lines earlier at :475, and it says "have the publisher sign with the same seed", describing the symmetric setup §4.3 explicitly rejected. The rewrite is in the remaining work; flagging it now so the name change isn't a surprise.

Logging asymmetry is intentional. Seeds are logged as a length only — not even a prefix, unlike the API token, because 4 hex chars of a 32-byte seed is 16 bits of a private key and there's nothing to gain from identifying it. Kids are logged in full, because "the operator deployed yesterday's key set" is otherwise invisible until it surfaces as an unexplained rejection.

Also fixed a pre-existing leak: --purge listed eventbridge-ntfy in neither the doomed preview nor the delete tuple, so a "purged" namespace kept an ntfy topic — a capability that lets anyone who learns it both read and publish your notifications.

Testing

568 passed, 5 skipped (5 cluster-gated). 17 new tests. ruff check and ruff format --check clean.

The three manifest tests this could have broken all still pass, and they're why the design looks the way it does: test_no_credential_is_ever_rendered_into_a_manifest (so the Secret is applied by the script, never declared — no secretGenerator), test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic (so enforcement stays out of the shared overlays), and test_no_manifest_sets_run_as_user (so no defaultMode: 0400, which would be root-only given OpenShift assigns the UID).

Verified by rendering rather than assumed:

  • data is still volume index 0 in all three existing overlays, so the demo overlay's op: replace on /spec/template/spec/volumes/0 still targets the PVC swap it means to.
  • No overlay renders a kind: Secret.
  • kind stays fully off; kind-signed renders enforcement plus both key paths, and those values map onto the real Cfg fields.
  • Keys from the generator verify through verify_with_keyset after a to_kafka_binary/from_kafka_binary round trip.

Not verifiable in my sandbox, and not claimed: kubectl apply --dry-run=client needs a reachable cluster to fetch the OpenAPI schema and fails even with --validate=false, and Kind can't run here (podman has no VM; creating one fails on the proxy's download cap). So schema validity of the new volume stanzas and the actual mount behaviour are unproven — covered by test_every_object_is_accepted_by_the_api_server when run against a real cluster.

Still to do on this branch

  • k8s_preflight.py check 14 — Secret present, keyset parses, every kid the ConfigMap names is in it
  • k8s_e2e_test.py — assert the terminal response carries ce_signature and the startup banner says signing is ON
  • k8s_demo_flow.py --forge — publish an unsigned event to the responses topic and assert it lands as phase="error" with the payload retained under data["rejected"]. Mechanism already validated against a live broker: kafka-console-producer.sh --property parse.headers=true yields a CloudEvent the consumer parses and the verifier rejects (binary mode matters — a structured-mode body gives attrs == {})
  • Rewrite README_PHASE1.md §"Signed events"

Question for review

Is kind-signed the right shape? The alternative is a --signed flag on the existing kind overlay. I went with a separate overlay so the unsigned path keeps working for S3/S5 and so the test overlay's no-credential assertion keeps meaning something — but the name is about to get baked into three more files (preflight, e2e, demo flow), so it's cheap to change now and annoying later.

Separately: S6 (#2596) specifies ES256 + a SPIFFE-derived ce-source + rotation, and this is Ed25519 + a static keyset. The eventing design docs never commit to ES256 — only the issue does, and §4.3 argues Ed25519 on merits with SPIRE as a later swap that leaves verification logic unchanged. Worth an explicit decision rather than a surprise at demo time; I'll comment on #2596.

Assisted-By: Claude Code

… keys

#879 wired signing in both directions but left it unreachable outside a laptop:
the ConfigMap shipped `*_VERIFY_KEYSET_PATH` and `*_SIGNING_KID` as empty strings
and said seed paths "belong in the Deployment beside the volume" — and no
Deployment had one. The only volume in either was `data: emptyDir`, so there was
no way to get a key into a pod. That blocks S6/S8/S9, each of which is defined as
"reproducible on a local Kind cluster".

* `scripts/gen_signing_keys.py` — no key generation existed anywhere, so enabling
  signing meant hand-rolling `os.urandom(32).hex()` and deriving public keys in a
  REPL. Seeds are written 0600 under a 0700 directory and **never printed**; only
  kids and public keys go to stdout, so a pasted terminal transcript cannot leak a
  private key. Public keys are derived from the seeds rather than typed, so the
  keyset cannot disagree with the keys it authorizes. Refuses to overwrite an
  existing seed without `--force`, since that key may already be deployed and
  approved.

* Both Deployments mount a seed Secret at `/etc/ce-signing` and the keyset
  ConfigMap at `/etc/ce-keyset`, both `optional: true` — without that, every
  signing-less overlay would block in ContainerCreating waiting for a Secret that
  will never exist. No `defaultMode`: 0400 would be root-only and there is no
  `runAsUser`/`fsGroup` (OpenShift assigns the UID), so the default 0644 is what
  makes the file readable.

* `k8s/base/keyset-configmap.yaml` ships **empty** and a test will keep it that
  way. "There is never a key in this file" is checkable; "only public keys in this
  file" is not, and a 32-byte seed is indistinguishable from a public key by length.

* `k8s/overlays/kind-signed/` flips enforcement on. A separate overlay rather than
  patching `kind`, because S3/S5 are explicitly unsigned milestones and the
  before/after contrast is the demo — and because the `test` overlay's
  "mounts no credential" assertion should keep meaning something.

**A per-service Secret, not one shared.** Caught while rendering the overlay: both
Deployments initially mounted the same `event-signing-key`, which would have given
the two services one seed. That collapses the asymmetry the design rests on —
EventBridge would hold the key it verifies runner responses with and could forge
them, which is the exact property that made HMAC unacceptable (DESIGN_PHASE2.md
§4.3). Now `eventbridge-signing-key` and `eventrunner-signing-key`. This deviates
from the single `event-signing-key` name in README_PHASE1.md:704-722; that section
is rewritten in a later commit, as it also still describes a symmetric setup.

Verified: `data` remains volume index 0 in all three existing overlays, so the demo
overlay's `/spec/template/spec/volumes/0` PVC patch still targets the right volume;
no overlay renders a `kind: Secret`; `kind` stays fully off while `kind-signed`
renders enforcement plus both key paths; the rendered env maps onto the real
config fields; and keys from the generator verify through `verify_with_keyset`.

551 passed, 5 skipped — including the three manifest tests this could have broken
(`test_no_credential_is_ever_rendered_into_a_manifest`,
`test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic`,
`test_no_manifest_sets_run_as_user`). ruff check and format clean.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…it on teardown

`k8s_deploy.py --signing-keys DIR` applies one seed Secret per service plus the
approved-key ConfigMap, modelled on `ensure_ntfy_secret`: via stdin so the seed
never reaches `ps` through argv, and a no-op when the flag is absent so a redeploy
cannot silently switch enforcement off and strand a signed topic.

It validates before applying, because every one of these mistakes produces a
cluster that refuses all traffic with no local symptom — the service starts, signs
under a kid nobody approved, and the other side rejects everything:

* the key directory and each named seed file must exist,
* `agents.json` must parse as a keyset and approve at least one kid,
* the kid each service will sign under must be IN that keyset.

Each failure names the file and the command that fixes it.

Seeds are logged as a length only — not even a prefix. 4 hex chars of a 32-byte
seed is 16 bits of a private key, and unlike an API token there is nothing to gain
from identifying it. The keyset gets the opposite treatment on purpose: kids are
logged in full, because "the operator deployed yesterday's key set" is otherwise
invisible until it surfaces as an unexplained rejection.

`--overlay kind-signed` without `--signing-keys` now aborts unless both Secrets are
already deployed. Both volumes are `optional: true`, so without this the pods would
start happily with enforcement on and no key, and refuse every event.

Teardown: `--purge` now deletes both signing Secrets and the keyset ConfigMap.
It also deletes `eventbridge-ntfy`, which was listed in neither the `doomed`
preview nor the delete tuple — so a "purged" namespace kept a capability that lets
anyone who learns it read and publish your notifications. Leaving private keys
behind is worse: a later deploy silently inherits an identity the operator thought
they had destroyed.

17 new tests covering generation and application, including that the two services
are never given the same seed (a shared one would let EventBridge forge any
agent's response — the HMAC property §4.3 rejected), that seeds land 0600 under a
0700 directory, that the CLI prints public keys but never a seed on either stream,
and that each malformed-key-directory case is refused before it reaches a cluster.

568 passed, 5 skipped. ruff check and format clean.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath mrsabath changed the title feat(eventing): make event signing deployable on a cluster feat(eventing): Phase 2 — make event signing deployable on a cluster Oct 5, 2026

@aslom aslom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 21d49c9 with design conformance as the primary lens. The diagnosis is
right and well evidenced — grep -rn "secretName\|configMap:" k8s/ returning nothing
against a ConfigMap that shipped *_KEYSET_PATH as empty strings is exactly the gap, and
configmap.yaml:54 really does promise a Deployment stanza that no Deployment had.

Design conformance: where it lines up

I checked every citation in the diff against the documents. They resolve, and the
reasoning is faithful:

Claim Document Verdict
seeds → Secret, keyset → ConfigMap DESIGN_PHASE2.md §5 ✔ exactly as written, including why the location is the control
one seed per service, never shared §4.3 — "EventBridge would hold the key it verifies with, so it could forge any agent's response" ✔ the strongest argument in the PR, and it is the design's own
Ed25519, not ES256 §4.3 ✔ grep -rniE "es256" over agentdocs/ returns nothing — your read is right, only the issue commits to it, and §4.3 names SPIRE as the upgrade that leaves verification unchanged
ER_REQUIRE_SIGNATURE, both *_KEYSET_PATH, both *_SIGNING_KID DESIGN_PHASE1.md §11 + DESIGN_PHASE2.md §5 ✔ every name matches, and all six op: replace targets exist in the base
EB_SIGNING_KID is the only kid accepted on group events §4.4 item 5 / §5 ✔ the overlay comment restates it correctly
bare §N = DESIGN_PHASE1.md pre-existing # §8.4 in eventrunner-deployment.yaml ✔ consistent with the house convention

Your three README claims are also accurate and, if anything, understated: :713 does say
event-signing-key, :710-716 does write a seed to /tmp and rm it against the
stdin-only rule at :474-477, and :715 does describe the symmetric setup §4.3 rejects.
It also says ER_VERIFY_KEY_PATH=/keys/seed.hex and mounts at /keys, so the mount path
drifts too.

One correction in your favour: the single-Secret convention you are deviating from is a
runbook, not a design.
DESIGN_PHASE2.md §4.4 and §5 — the authoritative signing
design — say nothing about Secret names or count. The only design-level sentence nearby is
DESIGN_PHASE1.md §11's "Ed25519 keys in a read-only Secret", which two Secrets satisfy
as easily as one. So the deviation costs a README rewrite you already owe, not a design
amendment. Worth saying in the body so it does not read as a bigger decision than it is.

Where it does not line up

Three places, one of them with teeth.

  1. The seed paths go into the ConfigMap, which configmap.yaml:54-55 says they must
    not
    — and the overlay quotes that sentence immediately before doing it. Not a leak
    (the value is a path), but it is a rule the repo states, the patch cites, and the patch
    breaks. It also collides with #883, open right now: that PR adds
    test_no_phase3_secret_is_inlined_into_a_manifest, asserting f"{var}:" not in rendered for seed-path variables. Its SECRET_PATH_VARS is scoped to Phase 3's three,
    so it does not fail today — but §8.3 states the rule as "continuing Phase 2 §5's", so
    ER_/EB_SIGNING_KEY_PATH joining that tuple is the obvious next edit, and it would
    fail against this overlay. Inline comment with both options.

  2. test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic now passes for
    a narrower reason than it documents.
    Its docstring says "NOTHING can inject a
    credential into these pods: no Secret is referenced" — and the test overlay now
    renders secretName: eventrunner-signing-key. Only the literal secretRef spelling in
    the assertion keeps it green. You cite this test as a constraint that shaped the design,
    so it is worth keeping it able to mean that.

  3. DESIGN_PHASE1.md §12's preflight table documents 12 checks; the script has 13
    (check_claude_cli_image), which IMPLEMENTATION_REPORT1.md:360,776 and
    README_PHASE1.md:189 all reference by number. §13 step 0 also still reads "checks 1-8,
    10-12". Pre-existing drift, not yours — but you are about to add check 14, so the
    §12 table and the §13 step list are two more lines for the same rewrite pass. Worth
    folding in rather than discovering later that the design documents 12 of 14.

Also: /etc/ce-signing, /etc/ce-keyset, seed.hex, eventing-keyset and the two
Secret names appear in no document in agentdocs/. That is fine for a draft, and your
README rewrite covers the runbook half — but note the feature's design lives in
DESIGN_PHASE2.md §4 while the rewrite target is README_PHASE1.md, and README_PHASE2.md
is already owed (it came up on #883). Putting the signed-deployment runbook into the Phase 1
document is the kind of choice that is cheap now and confusing in a month.

The must-fix

ensure_signing_material validates everything except the one thing that matters. It
checks the directory, that agents.json parses, that it approves at least one kid, that
each seed file exists, and that the kid is in the keyset — but never that the seed
derives the public key the keyset holds for that kid. A divergent pair passes every
check and produces precisely the cluster the function exists to prevent: pods start, both
volumes are optional: true, enforcement is on, and every event is refused with nothing
local to look at.

That state is reachable without hand-editing, which is the second finding: write_seeds
writes kid-by-kid and raises mid-loop on an existing seed, while agents.json is only
rewritten after it returns. Both reproduced below. Three lines in the deploy script
closes the whole class however the divergence arose — including the likeliest real case,
an agents.json copied from a teammate while the seeds are local.

Verification I ran

Check Result
python -m pytest in eventing/ @ 21d49c9 566 passed, 7 skipped (573 total — your 568/5 agrees; 2 extra skips are this sandbox's subprocess block)
17 new tests in test_signing_deploy.py confirmed
ruff check . / ruff format --check . @ 0.11.4 clean, 135 files formatted
kustomize build on all four overlays all four build
data is volume index 0 everywhere ✔ ['data', 'signing-key', 'keyset'] in all four; demo's index 0 is the PVC, so its op: replace still lands on data
no overlay renders kind: Secret ✔
kind-signed renders enforcement + both key paths ✔ all eight values, and they map onto real Cfg fields
kind stays fully off ✔ both flags "false", all four paths empty, no *_SIGNING_KEY_PATH key at all
CI 12/12 green
Commits 2, both Signed-off-by, Assisted-By not Co-Authored-By
.claude/ / .vscode/ changes none

I also applied both code suggestions to a scratch worktree: 566 passed, 7 skipped,
ruff still clean, and three tests written against the fixed behaviour pass — a
mismatched pair is now refused with nothing applied, a matching directory still applies
all three objects, and a refused regenerate leaves the directory byte-identical. Each
fails against its own revert.

kustomize build succeeding is weaker than kubectl apply --dry-run=server, which I
also cannot run — so your "schema validity of the new volume stanzas is unproven" stands.
It does rule out the failure mode I most expected: a replace op against a key that does
not exist in the base.

On your question

kind-signed as a separate overlay is the right shape. Two reasons beyond the ones
you give. A --signed flag on kind would have to mutate the ConfigMap at deploy time,
which puts enforcement outside the §14.1 manifest digest — the thing you correctly argue
the values belong inside. And the forged-event demo (S9) wants both overlays deployable
back to back in one session, which a flag on one overlay makes awkward. The name is fine;
kind-signed reads as "kind, plus signing" and that is what it is.

REQUEST_CHANGES for the missing seed/keyset agreement check only. Everything else is a
suggestion or a note on the rewrite you have already scheduled.

Comment thread eventing/scripts/k8s_deploy.py
Comment thread eventing/scripts/gen_signing_keys.py
Comment thread eventing/k8s/overlays/kind-signed/kustomization.yaml Outdated
Comment thread eventing/k8s/base/eventrunner-deployment.yaml
…d three more

Addresses the #884 review. All findings reproduced against `21d49c9` before fixing, and
each fix verified by reverting it.

**The must-fix: `ensure_signing_material` validated everything except the one thing that
mattered.** It checked the directory, that `agents.json` parses, that it approves a kid,
that the seed file exists, and that the kid is *in* the keyset — never that the seed
**derives** the public key the keyset holds for it. Reproduced:

    kid in keyset           : True
    seed file exists        : True
    keyset parses           : True
    pub(seed)               : 29acbae141bccaf0…
    keyset's pub for eb-01  : 03a107bff3ce10be…
    MATCH                   : False

A divergent pair passed every check and produced exactly the cluster the function exists
to prevent: pods start (both volumes are `optional: true`), enforcement is on, the
service signs with a key nobody approved, and every event is refused with nothing local
to look at. Now refused before anything is applied, with both fingerprints and the two
remedies named. A corrupt seed is a named check failure rather than a traceback.

**`write_seeds` is all-or-nothing.** It wrote kid-by-kid and raised mid-loop, while
`agents.json` is rewritten by the caller afterwards — so if the FIRST kid in sorted order
was missing and a later one present, the first got a brand-new seed and the keyset kept
the old public key for it. That is the one state the module docstring promises cannot
happen ("the keyset cannot disagree with the keys it is supposed to authorize"), and it
is reachable from a partial rotation or a restored backup. The pre-check now runs over
the whole set before any file is touched, and the refusal says "Nothing was written".

Both of these are the same hazard from opposite ends, which is why the deploy-time check
matters even with the generator fixed: the likeliest real case is neither, but an
`agents.json` copied from a teammate while the seeds on disk are local.

**The seed paths move out of the ConfigMap and into each Deployment's `env`.** The
overlay quoted `configmap.yaml:54-55` — "they name Secret mounts and belong in the
Deployment beside the volume" — immediately before doing the other thing. No leak, since
the value is a path, but a rule the repo states and the patch cites should not be a rule
the patch breaks. Followed the rule rather than rewriting it, which also keeps
`test_manifests.py`'s Secret-vs-ConfigMap assertions aligned and leaves #883 free to
extend its `SECRET_PATH_VARS` tuple to these two variables, as §8.3 implies it should.

`env` does not exist on either base container (only `envFrom`), so the patch adds the
whole list; an `env/-` append would fail at render time. All four overlays still build,
`kind` stays fully unsigned, and `kind-signed` renders both paths on the Deployments with
enforcement still in the ConfigMap where the §14.1 digest covers it.

**`test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic` passed for a
narrower reason than it documented.** Its docstring says "no Secret is referenced", and
the overlay now renders `secretName: eventrunner-signing-key` — only the literal
`secretRef` spelling kept it green. It now checks Secret references by name against an
allowlist of the two signing Secrets and rejects anything `anthropic`-shaped by either
spelling. Verified by injecting a credential Secret into the test overlay: the test goes
red, where before it would not have.

**§12 documented 12 preflight checks; the script has 13.** `check_claude_cli_image` was
added with the demo overlay and never got a table row, while
`IMPLEMENTATION_REPORT1.md` and `README_PHASE1.md` both reference checks by number. Added
row 13 with its reasoning (it runs on a cluster node because the CLI's bundled `bun`
aborts under QEMU user-mode emulation), and corrected "checks 1-8, 10-12" to "10-13" in
§13 step 0 and both places `k8s_deploy.py` repeats it. Pre-existing drift, folded in here
because check 14 is on this branch's own list.

Tests: 575 passing (+7), ruff clean, all four overlays build.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath

mrsabath commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

All four findings addressed in 803ccd9, each reproduced against 21d49c9 first and
each fix verified by reverting it. Thread replies have the detail; three things belong
here instead.

Your correction, accepted and now in the body. You are right that the single-Secret
convention I deviated from is a runbook, not a design — DESIGN_PHASE2.md §4.4 and §5
say nothing about Secret names or count, and DESIGN_PHASE1.md §11's "Ed25519 keys in a
read-only Secret" is satisfied by two as easily as by one. I had written it up as a
bigger decision than it is. It costs the README rewrite I already owe, not a design
amendment, and the PR body says so now.

§12/§13 preflight drift, folded in. You were right that this is pre-existing and
right that it was mine to fix anyway, since check 14 is on this branch's list and
discovering the design documents 12 of 14 later is worse. Added row 13 for
check_claude_cli_image with its reasoning — it runs on a cluster node because the CLI's
bundled bun aborts under QEMU user-mode emulation, so a cross-built image cannot execute
it on an arm64 builder — and corrected "checks 1-8, 10-12" to "10-13" in §13 step 0 and
both places k8s_deploy.py repeats it.

On the runbook target, which I think you are right about and I want to flag rather than
quietly decide.
You note the feature's design lives in DESIGN_PHASE2.md §4 while the
rewrite target is README_PHASE1.md, and that README_PHASE2.md is already owed. I agree,
and it is the better home: a signed-deployment runbook in the Phase 1 document is exactly
the kind of filing that is cheap now and confusing in a month.

So the plan is README_PHASE2.md carrying the signed-deployment runbook, rather than
rewriting README_PHASE1.md §"Signed events" in place — which also means the undocumented
names you listed (/etc/ce-signing, /etc/ce-keyset, seed.hex, eventing-keyset and
the two Secret names) land in the Phase 2 document where the rest of their design is.
README_PHASE1.md §"Signed events" then gets a pointer and loses the three wrong claims
you confirmed, rather than a rewrite.

That makes the remaining work on this branch:

  • k8s_preflight.py check 14 — Secret present, keyset parses, every kid the ConfigMap
    names is in it and its seed derives the approved key (the must-fix generalised
    to preflight, where it catches the problem before deploy rather than during)
  • k8s_e2e_test.py — assert the terminal response carries ce_signature and the
    startup banner says signing is ON
  • k8s_demo_flow.py --forge
  • README_PHASE2.md with the signed-deployment runbook; README_PHASE1.md
    §"Signed events" reduced to a pointer with its three wrong claims removed
  • §12 table row 14

On kind-signed as a separate overlay — thank you for the two extra arguments, both
of which are better than mine. The §14.1 digest one in particular: a --signed flag would
have to mutate the ConfigMap at deploy time, putting enforcement outside the digest,
which contradicts the reason I argued the values belong in the manifest. Name and shape
stay.

On ES256 vs Ed25519 — your grep -rniE "es256" over agentdocs/ returning nothing
confirms the read, so I will comment on #2596 proposing the design documents win and the
issue is amended, rather than changing the implementation. §4.3 argues Ed25519 on merits
and names SPIRE as the later swap; switching to ES256 to match an issue nobody has
defended on technical grounds would be the wrong direction.

Tests: 575 passing (+7), ruff clean, all four overlays build.

@aslom aslom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 803ccd9. All four round-1 findings are fixed, and I checked each against
the tree rather than against the replies — including mutation-reverting every fix.

# Finding Fix Verified by
1 ensure_signing_material never checked pub(seed) == ks[kid] the check, with both remedies in the message mutation: 4 failures; my own round-1 repro now refuses the mismatched pair with nothing applied
2 write_seeds could leave a new seed beside the old public key whole-set pre-check, "Nothing was written" mutation: 2 failures (one more than you claimed); ran the first-kid-missing case — seed not re-created, agents.json untouched
3 seed paths in the ConfigMap contradicted configmap.yaml:54-55 moved to container env on both Deployments rendered all four overlays: no overlay has a *_SIGNING_KEY_PATH key in eventing-config, and kind-signed carries both on the Deployments
4 the test-overlay assertion passed for a narrower reason than it documented secretName: allowlist by name, plus an anthropic-shaped reject read; see the note below on what exercises it

Two things you did beyond what I asked, both right:

  • load_seed rather than bytes.fromhex, so a corrupt or wrong-length seed is a
    named check failure instead of a traceback — and it means a public key accidentally
    placed in seed.hex is caught by the derivation check rather than silently decoding.
  • DESIGN_PHASE1.md §12 now documents check 13 and §13 step 0 reads "checks 1-8,
    10-13". That was in my review body with no commentable line, so thank you for picking
    it up from prose.

Your reasoning on finding 3 is also the one I'd have argued for: following the rule costs
one extra patch block, while changing it would have meant coordinating a carve-out across
two open PRs and three statements of the same sentence. And you are right that #883
needs no change
as a result — I re-checked, its SECRET_PATH_VARS tuple can be extended
to ER_/EB_SIGNING_KEY_PATH freely now and this overlay stays green.

Verification I ran

Check Result
python -m pytest in eventing/ @ 803ccd9 573 passed, 7 skipped (was 566/7)
ruff check . / ruff format --check . @ 0.11.4 clean, 135 files formatted
kustomize build on all four overlays all four build
Seed-path keys in any rendered eventing-config none, in all four
kind-signed Deployment env EB_SIGNING_KEY_PATH / ER_SIGNING_KEY_PATH, enforcement still in the ConfigMap where §14.1's digest covers it
kind still fully unsigned both flags "false", all paths empty, no env entries
demo's three env entries intact yes — see the one nit below, which is about a future overlay, not this one
My round-1 fixed-behaviour tests 3/3 pass
New tests in test_signing_deploy.py 17 → 23
Commits 3, all Signed-off-by, Assisted-By not Co-Authored-By
.claude/ / .vscode/ changes none

Ready to merge? The code is. CI is not — and it is not your fault

Almost nothing has actually run on 803ccd9, and what shows as failed was cancelled
rather than failing.
Job-level detail from the Security Scans run (37366549985):

trivy-scan         cancelled   19:55:44Z -> 20:27:32Z   failed steps: (none)
hadolint           cancelled   19:55:44Z -> 20:28:09Z   failed steps: (none)
codeql             cancelled   19:55:44Z -> 20:30:12Z   failed steps: (none)
dependency-review  success     20:08:51Z -> 20:08:58Z

Three jobs sat queued for ~32 minutes and were cancelled with zero failed steps — they
never executed. dependency-review ran as soon as a runner freed up and passed in 7 s. The
whole CI workflow (lint, test, eventing-test, test-startup) is still queued
~35 minutes after creation, and PR Verifier recorded startup_failure twice. hadolint
and trivy-scan passed on 21d49c9 in 7 s and 33 s, and this PR touches no Dockerfile and
adds no dependency.

So the gate is unresolved, not red: this is runner starvation, not a result. Re-trigger the runs and confirm 12/12 before
merging
— an empty push or closing and reopening the draft will do it. If eventing-test
comes back failing after a genuine run, that would be new information and worth a look,
but locally it is 573 passed with the same pytest and kafka-python CI installs.

One more gating item, unrelated to CI: this is still a draft, so the merge decision is
yours either way. If you want the remaining four items on your own list (preflight check
14, the e2e assertions, --forge, the README_PHASE1 rewrite) in this PR rather than a
follow-up, say so and I will hold the approval for them; I would rather see them land
separately with their own tests.

Approving the code. Both must-fix mechanisms are closed and reproduced closed, the
three suggestions are implemented, the design-conformance contradiction is resolved in the
direction that keeps all three statements of the rule aligned, and nothing regressed. The
nit below is about an overlay that does not exist yet.

Comment on lines +71 to +73
# `env` does not exist on the base container (only `envFrom`), so this adds the
# whole list rather than appending to it. A trailing-slash `env/-` op would fail
# with "path does not exist" at render time.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This comment warns about the failure that is loud and not about the one that is
silent — which matters because the PR plans to copy this patch into three more files.

env/- failing with "path does not exist" is correct and you are right to record it. The
hazard worth the comment is the other direction: op: add on /…/containers/0/env
replaces the list when one already exists.
It is safe here because kind-signed
builds on ../kind, whose containers have no env — I rendered all four overlays to
confirm:

kind         eventrunner  env=[]
kind-signed  eventrunner  env=[('ER_SIGNING_KEY_PATH', '/etc/ce-signing/seed.hex')]
demo         eventrunner  env=[('ER_MOCK_CLAUDE','false'), ('HOME','/data/home'),
                               ('CLAUDE_CONFIG_DIR','/data/home/.claude')]

demo is the one that has entries. I built a throwaway demo-signed overlay by changing
nothing but the base — - ../kind → - ../demo — with this patch block verbatim:

hypothetical demo-signed eventrunner env = ['ER_SIGNING_KEY_PATH']
demo's own                              = ['ER_MOCK_CLAUDE', 'HOME', 'CLAUDE_CONFIG_DIR']
-> demo's three entries are LOST, silently

kustomize build succeeds, nothing warns, and ER_MOCK_CLAUDE=false is gone — so a
demo-signed built the obvious way would quietly run the demo in mock mode, which is
precisely what test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic
exists to keep honest at the other end.

Not a defect in this PR, and I am not asking for a demo-signed overlay. Just the half-
sentence, since whoever writes it will start from this file:

Suggested change
# `env` does not exist on the base container (only `envFrom`), so this adds the
# whole list rather than appending to it. A trailing-slash `env/-` op would fail
# with "path does not exist" at render time.
# `env` does not exist on the base container (only `envFrom`), so this adds the
# whole list rather than appending to it. A trailing-slash `env/-` op would fail
# with "path does not exist" at render time.
#
# NOTE for anyone basing a signed overlay on `demo` instead: `add` on this path
# REPLACES an existing list rather than extending it, and silently — `demo`'s
# container has three entries (`ER_MOCK_CLAUDE=false`, `HOME`,
# `CLAUDE_CONFIG_DIR`), and a copy of this block drops all three while
# `kustomize build` still succeeds. `ER_MOCK_CLAUDE=false` going missing puts the
# demo back in mock mode. From `demo`, append with `op: add` on `env/-` per entry.

Comment on lines +130 to +134
allowed_secrets = {"eventbridge-signing-key", "eventrunner-signing-key"}
referenced = set(re.findall(r"secretName:\s*(\S+)", doc))
assert referenced <= allowed_secrets, \
f"unexpected Secret reference: {sorted(referenced - allowed_secrets)}"
# No credential Secret, by either spelling, and nothing credential-shaped.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This is the right assertion and nothing automated runs it.

The allowlist-by-name is what I was asking for — secretName: only, so the signing-key
volume name is not mistaken for a Secret reference, and that trap is worth the comment
you put on it. I could not mutation-verify it, though: the test is @needs_kubectl and
skips without a cluster, so it is 1 of the 5 no reachable cluster skips both here and in
CI. Reverting the three new lines leaves tests/test_manifests.py at 29 passed, 5 skipped
— unchanged.

Your manual check (patching a credential Secret into the test overlay and watching it go
red) is the right verification and I believe it; I am noting the gap rather than disputing
the fix, because this is the same shape as IMPLEMENTATION_REPORT3.md §2's note about the
claude --help skip: a promise that currently only holds on a laptop.

render() is the thing that needs kubectl, and kustomize is a separate binary that
this sandbox has — so if render() could fall back to kustomize build when kubectl is
absent, this test and the four other manifest assertions would start running in CI with no
cluster. That is a bigger change than this PR should carry; worth an issue alongside
#885-#889 so it is not rediscovered the next time a manifest assertion silently stops
meaning anything.

@aslom aslom closed this Oct 5, 2026
@aslom aslom reopened this Oct 5, 2026
@mrsabath
mrsabath marked this pull request as ready for review October 5, 2026 21:26
@mrsabath
mrsabath requested a review from a team as a code owner October 5, 2026 21:26
mrsabath added a commit that referenced this pull request Oct 5, 2026
…and how to check them (#890)

* docs(eventing): record the claims a docs review disproved, and how to check them

The user-facing eventing pages (rossoctl/rossoctl#2609) restated every claim in
DESIGN_PHASE2 for a reader who cannot see the code. Six review rounds then ran
each one against eventing/ at d9677dd. Thirteen did not hold.

DESIGN_PHASE2 §8 records them, rather than editing §§2.6, 3, 4.3 and 4.4 in
place, so the reasoning stays readable as what was intended and §8 says what the
code does. The four §4.4 promises that fail: audit mode logs nothing, a pinned
group event is flagged and then applied anyway, a terminal signature stops
proving who finished a run once INSERT OR REPLACE lets an unsigned frame
overwrite it, and nothing checks that EB_SIGNING_KID is in the keyset. A fifth,
promised nowhere and worse: EB_REQUIRE_RESPONSE_SIGNATURE with no keyset refuses
nothing, silently, while the request side with no key refuses everything.

CLAIMS.md turns that into five questions to ask of a control claim before it
ships — which code path, what default, what happens half-configured, can an
attacker choose the input the check reads, what does it become one release from
now — plus how to run the check cheaply and what a correction should preserve.

None of the thirteen was careless about mechanism; §4.4 is unusually precise.
Each was written from the change that introduced the control and stayed true for
the configuration that change was exercised in. Every one coexisted with a
passing test suite.

Tracked as #885, #886, #887 and #888; #889 came out of the same review without
being a claim in any document.

Docs only. No code changes, so this does not overlap #884, and it touches
agentdocs/README.md where #883 also does — whichever merges second resolves one
table row and one reading-order entry.

Assisted-By: Claude Code
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* docs(eventing): fold CLAIMS.md into DESIGN_PHASE2.md §8, per review

@aslom on #890: "please merge CLAIMS into DESIGN_PHASE2.md or DESIGN_PHASE3.md as I
think the claims are part of design and they should reflect the phase of the design
they are related to."

Taken. `CLAIMS.md` is gone and its content is now §8.5–§8.8 of `DESIGN_PHASE2.md`,
directly after the thirteen findings it was derived from. Phase 2 is the right home: the
findings are claims *this* document made, and §8.4 already handed off to the procedure,
so the two halves were a section boundary apart for no reason.

The content moved rather than being rewritten — 133 lines out, 135 in. Headings demote
one level (the five questions become §8.5.1–§8.5.5) and the three references that
assumed a separate file are re-pointed: "see its §8 for the list" is now "see above",
and "`DESIGN_PHASE2.md` §8 is that shape" is now "§8.1–§8.3 above are that shape".

**On the phase question, which is the substance of the request.** The findings are
Phase 2's and the procedure is not, so §8 now opens by saying which is which:

> §8.1–§8.4 are about **Phase 2** specifically: thirteen claims *this document* made
> about the identity and signing controls, which do not hold in the code. §8.5–§8.8 are
> the general procedure that came out of them, and apply to a claim made in any phase —
> Phase 3's design makes claims of the same shape about tenancy and the signed attribute
> set, and nothing here is Phase 2 only.

That answers the question the PR body asked out loud ("is `CLAIMS.md` the right home? It
is process guidance sitting among design and build records") rather than leaving the
ambiguity in place. §8.5's own lead-in says the same thing for a reader who arrives
there directly, and gives the reason it is filed here rather than at repo root: a
procedure kept away from the findings that motivated it is a procedure nobody reads.

`agentdocs/README.md`: the standalone index row is gone, the Phase 2 row now mentions
both halves of §8, and the reading-order entry reads "**Writing a claim about a control,
in any phase?**" pointing at §8.5 for the questions and §8.1–§8.3 for the findings —
with "The findings are Phase 2's; the questions are not" stated explicitly, since that
distinction is the whole point of where this now lives.

Verified every `§N.N` reference in the merged document resolves to a heading that
exists, and that no `CLAIMS` reference survives anywhere.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

---------

Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath
mrsabath merged commit a6b22b7 into main Oct 5, 2026
18 of 27 checks passed
mrsabath added a commit that referenced this pull request Oct 5, 2026
…kind-signed

Follow-up to rebasing onto `main` now that #884 and #890 have merged. Both predicted
conflicts were in `agentdocs/README.md` and resolved as planned — `main`'s Phase 2 row
(which gained §8 from #890) plus this branch's Phase 3 row. `test_manifests.py`
auto-merged, keeping #884's `allowed_secrets` fix beside this branch's Phase 3
classification.

That left two things the merge did not resolve by itself.

**Extending `SECRET_PATH_VARS` to #884's seed paths needed the rule split, not widened.**
#884's review predicted this tuple would be extended to `ER_/EB_SIGNING_KEY_PATH` and
checked that doing so stays green. It does for the ConfigMap half and would NOT for the
literal-value half, because those two variables hold a **mount point, not key material** —
and #884 deliberately sets them as container `env` entries so the path sits beside the
volume that supplies the seed, which is what `configmap.yaml` asks for.

So the tuple is now two:

- `SECRET_VALUE_VARS` — the three whose *value* is key material. No literal value
  anywhere, which is the original assertion.
- `SEED_PATH_VARS` — #884's two. A literal value is correct; a ConfigMap key is not.

Both still obey "never from `config.toml`", so that check covers all five now where it
covered three. Treating a path naming a secret as if it were the secret would have forced
#884 to either revert its own correct fix or carve itself out of the rule — the distinction
is the thing worth encoding.

**`kind-signed` was not in `OVERLAYS`.** #884 added the overlay and every assertion in
this file silently skipped it — including the seed-path rule the PR exists to respect, the
`replicas` invariants, and the no-credential checks. Added, which is also what makes the
new assertion meaningful: with `kind-signed` absent, injecting a seed path back into
`eventing-config` passed; with it present, that regression fails as it should. Verified
both ways.

Tests: 899 passing (875 here + #884's 24 picked up in the rebase), ruff clean, all four
overlays build.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
mrsabath added a commit that referenced this pull request Oct 5, 2026
The remaining nit from the #884 approval, which merged before it was addressed.

My comment on both patch blocks warned about the failure that is **loud** — an
`env/-` append failing with "path does not exist" when no list exists — and not about
the one that is **silent**: `op: add` on `…/containers/0/env` *replaces* the list when
one already exists.

Safe in `kind-signed`, which builds on `../kind` whose containers have no `env`. Not safe
in the overlay the PR plans to copy this patch into. Reproduced by building a throwaway
`demo-signed` that changes nothing but the base:

    demo-signed  eventrunner env = ['ER_SIGNING_KEY_PATH']
    demo's own                   = ['ER_MOCK_CLAUDE', 'HOME', 'CLAUDE_CONFIG_DIR']
    -> all three LOST, silently

`kustomize build` succeeds and nothing warns. Losing `ER_MOCK_CLAUDE=false` means a
`demo-signed` overlay would run the demo in **mock mode** — which is exactly what
`test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic` exists to keep
honest at the other end, so the failure would be invisible from both directions.

The comment now says that, and names the fix for whoever writes that overlay: `op: add`
on `env/-` per entry, against a base that already has the key.

Reviewer's words, and the right call to spend half a sentence on — this file is the
starting point for three more (preflight, e2e, demo flow) by the PR's own plan.

575 passing, `kind-signed` still builds, no behaviour change.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
mrsabath added a commit that referenced this pull request Oct 5, 2026
…s, declarative agents (#883)

* feat(eventing): Phase 3 step 1 — tenancy keys and declarative agent specs

DESIGN_PHASE3.md T1 and T13, the two tasks with no dependencies. Nothing here
changes behaviour: `single` tenancy is the default everywhere and the `default`
AgentSpec reproduces Phase 2's argv byte for byte.

shared/tenancy.py (T1)
  `userkey(issuer, userid)` is the only place an identity becomes a name. The
  slug is deliberately lossy so keys stay readable in `kubectl get kafkatopics`;
  that is safe only because 32 bits of SHA-256 over the canonical form are
  appended, so two identities that slug identically still differ. The issuer is
  inside the hashed bytes as well as the prefix — without it, configuring a
  second issuer is a way to impersonate an identity on the first. The digest
  input is joined with \x1f so the issuer/userid boundary is unambiguous.

  Canonicalisation is per-issuer and minimal: GitHub folds case (and must agree
  with `ghauth.is_allowed`, which a test pins), email lower-cases only the
  domain per RFC 5321, static is byte-exact. Erring toward two tenants for one
  human is wasteful; the other direction is a leak.

  `TopicSet` is the one place §3.1's layout is written down, and refuses to name
  a topic without a userkey in multi mode rather than falling back to a shared
  one. `ntfy_topic` is HMAC-derived so the name carries no identifier (§4.1).

eventrunner/agentspec.py (T13)
  One TOML file per agent, carrying the tool policy that §7.4 depends on — which
  is why §9 puts this before triggers. Validation refuses rather than coerces: a
  policy that silently drops the clause it could not parse would hand a
  trigger-driven agent the tool the operator meant to remove. A missing
  non-default agent is an error event, never a silent fallback to `default`.

  `build_cmd` takes a spec. The request still wins on `max_turns`/`model` (Phase
  0 wire parameters), but the tool policy, permission mode and system prompt are
  spec-only: a request that could widen the sandbox is a request that can escape
  it.

Tests: 105 new, 656 total passing, no existing test modified. The argv-equality
test compares against a literal transcription of Phase 2's `build_cmd` rather
than a call into the current one, so it catches a regression instead of
following one.

The `claude --help` flag assertion covers only the flags Phase 3 adds.
`--max-turns` is supported but undocumented in `--help` on 2.1.270, so asserting
the pre-existing flags would fail against a working deployment.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* feat(eventing): Phase 3 steps 2-4 — TopicSet wiring, user registry, signed tenancy

DESIGN_PHASE3.md T2, T3, T4 and T22. Still no behaviour change in the default
configuration: `EB_TENANCY_MODE=single` is the default, and the 656 tests that
existed before this branch all pass untouched. That is the property §9 asks steps
1-4 to preserve — refactors with tests, everything risky behind a flag that is off.

T2 — TopicSet threaded through (§3.2)
  One set, built once in `__main__` and handed to the producer, the responses
  consumer, both mirrors and the config, so §3.1's layout appears exactly once.
  `publish_request` takes a `userkey` and resolves the topic from it; one producer
  can send to any topic, so there is no second producer and no connection per user.

  The explicit-topic-list option, not `subscribe(pattern=...)`. A pattern is picked
  up by metadata refresh (5 min by default), so a new user's first response can be
  published before anyone is subscribed and with `auto_offset_reset=latest` it is
  never read — the page stays empty and no error appears anywhere. The blocking
  `ensure_subscribed` that closes the remaining race is T6.

  GroupMirror takes the full topic list directly: it is one-shot at startup, so
  there is no new-user race to lose, and a tenant whose topic is not provisioned
  yet is skipped with a log line rather than costing everyone their group history.

T3 — the registry and Caller (§2.4, §2.5)
  `Caller` carries the tenancy key so no caller derives it twice. `submitter_iss`
  is deliberately not the same as `issuer`: a static identity needs an issuer for
  its key to separate from a GitHub login of the same name, but Phase 2 made an
  absent `ce_submitteriss` mean "an operator typed this", and promoting it would
  upgrade an assertion into a verified claim.

  The registry refuses rather than skips — the opposite of `auth.parse_tokens`,
  because a skipped user is one whose topics exist and whose events have nowhere to
  go. A recorded userkey that disagrees with this build's derivation is a startup
  refusal naming both values, and two identities colliding on one key is refused,
  which is what covers §2.3's 32-bit birthday bound.

  `multi` mode refuses anonymous with 401 and an unregistered user with 403. Both
  disagree with the single-tenant defaults on purpose; `single` is byte-identical
  to Phase 2.

T4 — SIGNED_ATTRS += userkey, depth (§8.3)
  One change, both attributes, because adding one changes canonicalisation and a
  signer and verifier on different versions disagree about every signature. Both
  have to be covered rather than merely present: a mutable `userkey` lets anything
  with topic write access file events into another user's history, and a resettable
  `depth` is not a hop limit.

T22 — the Secret-vs-ConfigMap rule, pinned in test_manifests.py
  The secret-path variables are declared ahead of the tasks that read them (T8-T16)
  so the guard is in place when they land. The classification test only asserts the
  ConfigMap side against live code, and says so, rather than looking stronger than
  it is.

Known gap, stated rather than hidden: `/continue` resolves the owning tenant from
the recorded submitter so a resume reaches the right runner, but does not yet check
that the caller IS the owner — owner-scoped reads and `?k=` keys are §6.2/§4.4
(T7/T9, step 5). Until those land, `multi` mode's `/continue` is as open as Phase
2's. The lookup also cannot distinguish a GitHub `alice` from a static `alice`;
`multi` mode refuses both anonymous and static submissions, so GitHub is the only
issuer that reaches it today, and this moves to T5's global index when that exists.

Tests: 66 new (722 total), ruff clean. Verified both services start in every
tenancy configuration and that all four refusals print readable messages rather
than tracebacks.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* docs(eventing): record which of Phase 3 is implemented and which is still design

The Phase 3 design doc said "nothing in this document is implemented yet" and the
agentdocs index said "Design only — not implemented". Both are now wrong in a way
that matters: a reader who trusts them will either re-implement steps 1-4 or assume
`EB_TENANCY_MODE=multi` gives them isolation it does not yet give.

So the status lines name the tasks that landed (T1, T2, T3, T4, T13, T22 = §9 steps
1-4) and say plainly that everything from step 5 on — per-user stores, owner-scoped
reads, transcript auth, `k8s_tenant.py`, ntfy isolation, triggers, fetched skills,
Kafka ACLs — is still design only.

The component README gains a Phase 3 configuration table covering only the six
variables whose code exists, with two notes that are the point of including it at
all: `multi` mode is not yet isolation anyone should present as such (the HTTP reads
and `/continue` are still as open as Phase 2's, because steps 5-8 are not in), and
adding a user is an operator action because EventBridge has no Kubernetes client.

Verified against the code rather than assumed: only `POST /v0/agents` and
`POST /v0/groups` call the auth path, so every GET, `/continue` and `/transcript`
is open in both modes today.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): refuse an unresolvable /continue with 503 instead of crashing

Found reviewing the previous commit. In `multi` mode `_owner_userkey` returned
`None` both when no key was needed (single mode) and when the owning tenant could
not be determined. The second `None` flowed into `publish_request` ->
`TopicSet.requests(None)`, which raises `ValueError` by design — publishing a resume
to a shared topic would run it on another tenant's runner, under that tenant's
credential — but nothing caught it, so the request died as a WSGI 500 with a stack
trace instead of a status code.

Three ways to reach it, none of them abuse:

  * a user removed from the registry;
  * a `prompts` row whose `submitter` is NULL — the column is nullable and pre-dates
    auth, so these rows exist in any store that outlived Phase 2;
  * a correlation back-filled from the topic by `RequestsMirror`, which records no
    submitter at all.

`_owner_userkey` now returns `(userkey, unresolved_reason)` with exactly one set, so
the two `None` cases are distinguishable, and both `/continue` paths check it before
writing anything — no touched session row and no orphan prompt row claiming a turn
that never ran. The JSON route answers `503` naming what could not be resolved,
matching how §2.5 treats a user whose topics do not exist. The HTML form route
answers `503` with a plain-text body rather than redirecting: a silent `303` back to
the transcript page would look like the turn was accepted and then vanished, which is
the Phase 2 §6.1 symptom class this phase keeps trying to avoid.

Only reachable with `EB_TENANCY_MODE=multi`, so the default path was never affected.

13 new tests (735 total). Verified they are load-bearing by reintroducing the bug:
the four refusal tests fail without the guard and pass with it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* feat(eventing): Phase 3 T5 — per-user stores and the global ownership index

DESIGN_PHASE3.md T5 (§6.1, §2.6), the keystone the rest of step 5+ depends on: T7
(owner-scoped reads), T8 (transcript auth), T15 (triggers) and T21 (deletion) are all
blocked on it. `EB_TENANCY_MODE=single` remains the default and reproduces Phase 2.

eventbridge/owner_index.py — the one deliberately cross-tenant table
  `correlationid -> userkey`, UNIQUE on `correlationid` **alone**. That is the part to
  get right: `(userkey, correlationid)` is the correct primary key inside a tenant's
  store and exactly the wrong constraint here, because two tenants minting one id
  produce two distinct tuples, never conflict, and the collision passes through
  silently — which is the failure the guarantee exists to prevent.

  It has to be cross-tenant: the lookup that decides which tenant owns a correlation
  must happen before a per-user store can be chosen. It is also what lets
  `sessionuuid = uuid5(NAMESPACE, correlationid)` stay UNSALTED (§2.6) — salting it
  would break every existing session for a collision this constraint removes.

  `owner_of` returns `(userkey, known)` because a bare `None` conflates "the shared
  tier owns this" with "never seen", and §6.2's 404 rule needs them distinguished.

eventbridge/store_registry.py — per-user stores, LRU-bounded
  Separate FILES, not one database with a `userkey` column. `store.py` has 28 methods
  and every one touches tenant data; a `WHERE userkey = ?` remembered 28 times is one
  that will eventually be forgotten, with a user reading someone else's conversation
  as the symptom. Separate files make the mistake structurally unavailable — the
  connection IS the tenant.

  Two connections per tenant in WAL mode (main + -wal + -shm) bounds this near 150-200
  tenants under a 1024 soft RLIMIT_NOFILE, hence `EB_MAX_OPEN_STORES=64`. A store with
  live SSE subscribers is PINNED: evicting one makes an open page stop updating with no
  error anywhere. If everything is pinned the cache exceeds its ceiling rather than
  breaking a viewer — over-budget degrades and is visible on /healthz; a dead stream
  does not recover.

Minter now consults the index (§2.6)
  One indexed SELECT per mint replaces the startup seeding loop. With N per-user stores
  the old loop meant N SQLite opens before the socket binds — 100 tenants x 10,000
  correlations for a guarantee one SELECT gives directly. The claim happens inside the
  retry loop, so a candidate lost to a concurrent worker retries instead of both callers
  believing they own it.

  `mint_for()` passes `userkey` only to minters that accept it, by signature inspection
  rather than by catching TypeError around the call — a working `mint()` can raise
  TypeError from its own body, and swallowing that would silently drop the ownership
  claim.

Routing
  Responses are filed by the event's OWN signed `ce_userkey`. One with no userkey in
  multi mode lands in `shared/` and increments `unattributed` (now on /healthz): never
  guessed into a tenant's store, never dropped. `/continue`, the reads, the group routes
  and the deadline sweeper all route through the index — the sweeper across every tenant,
  because a stalled batch belonging to an untouched tenant is exactly what deadlines are
  for and also the least likely to be in the LRU.

  This retires the submitter-derived owner lookup from the previous commit, and with it
  the limitation that it could not tell a GitHub `alice` from a static `alice`.

Single-tenant mode keeps its files exactly where Phase 2 left them — directly in the
bridge root, not under `shared/` — so an upgraded deployment's sessions and transcripts
keep showing up in the UI instead of appearing to vanish.

Two bugs found and fixed while building this:

  * **Eviction could return a closed store.** With every older entry pinned, the LRU
    took the store it had just opened and handed the caller a closed one — surfacing
    later as `ProgrammingError: Cannot operate on a closed database` from somewhere that
    looks unrelated to caching. Fixed with `protect=`, with a regression test.
  * **`except Exception` hid a signature mismatch.** Adding `userkey=` to
    `publish_group_event` made every group event stop publishing, and the broad handler
    around it turned that into a batch that silently never completed. TypeError now
    re-raises at both call sites.

`Store.close()` added (it had none), and it wakes subscribers first so a viewer blocked
on `Event.wait()` notices shutdown instead of hanging to its own timeout.

Tests: 49 new (785 total), ruff clean. Verified both modes start and that the on-disk
layout is upgrade-safe.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): evict stores by dropping the reference, not by closing them

Found reviewing T5. `StoreRegistry` eviction called `Store.close()`, which created a
use-after-close race on every multi-call read path: a handler resolves a store, and
while it is still making calls a *concurrent request for a different tenant* pushes
the cache over `EB_MAX_OPEN_STORES` and closes it underneath. The next call fails with

    sqlite3.ProgrammingError: Cannot operate on a closed database

from a line that looks nothing to do with caching. `get_html` alone makes five calls on
one store, so the window is wide rather than theoretical, and reproducing it needs only
`max_open=1` and two tenants.

Eviction now drops the registry's reference and lets CPython close the connections when
the last holder goes away — exactly the lifetime that is safe. File descriptors are
reclaimed slightly later than an explicit close; that is the right trade, because the
ceiling is a soft budget while a closed connection under an in-flight request is a 500.

`_sse_generator` additionally now subscribes BEFORE the replay read. Pinning is what
keeps a subscribed store in the cache so later readers get the same object and see the
subscriber's notifications, and the old order left the store unpinned between
`_store_of()` and `subscribe()`. Reordering is harmless: the Event only ever means
"there may be something new", every read is `since_seq=sent`, and an already-set Event
costs one extra query.

Worth noting what the first fix attempt got wrong, since it is the more interesting
mistake: pinning only the SSE path would have closed the case that was easiest to see
while leaving every ordinary read exposed. The bug was in the eviction contract, not in
one caller.

Two regression tests, one asserting an evicted-but-held store stays usable for both
reads and subscribe, one pinning the subscribe-before-replay order. 787 passing.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): teach GroupMirror about tenancy, and stop hiding index failures

Three bugs from an adversarial review of T5. All three were confirmed by driving the
real code, and all three are multi-tenant-only — the default `single` path is unaffected.

**1. Restart-orphaned groups were never settled (`_settle`).** It called
`maybe_complete(gid)` with no `userkey`, so it looked in the SHARED store for a group
that lives in a tenant's store, found nothing, and returned False. Every group a restart
left unfinished stayed unfinished forever, and the failure mode is silence — which §21.6
calls the worst one. The mirror now resolves the owner per group from the ownership index.

**2. Replayed member events landed in the shared store.** `on_member_event` reads
`ce_userkey` off the event, and any response published *before* Phase 3 has none — so
member rows went to `shared/` while the group row sat in the tenant's store, leaving the
group's counts permanently wrong and `_decorate_member` reading from a third place.
Pre-existing Kafka history is exactly what this mirror exists to replay, so that is the
normal upgrade path, not an edge case.

The mirror now back-fills a missing `userkey` from the index. It only ever FILLS: an event
carrying its own `userkey` keeps it, because that attribute is signed (§2.6) and the index
is not authoritative over a signed assertion.

Both of these had one root cause — the mirror knew about per-user topics but not about
which tenant a replayed group belonged to.

**3. A broken index surfaced as "correlation ID space exhausted".** `mint()` caught
`Exception` around `claim` to treat a lost race as a retry, so a genuinely broken index
(disk full, database locked, schema missing) was retried 2000 times and then reported a
message that sends the reader to the word lists rather than the database. Now catches
`Collision` specifically; everything else belongs to the caller.

Also closes a test-strength gap the review found: deleting the `_has_subscribers` pinning
check entirely still passed every `test_store_registry.py` test. The existing tests
asserted object identity and counters, not the notification path that is the actual reason
for the rule — now that eviction only drops a reference, a held store keeps working, so
what actually breaks is that the *writer* gets a new Store whose subscriber map is empty
and the viewer is never notified. The new test asserts that end to end; removing the
pinning check now fails 4 tests instead of 0.

Tests: 8 new (795 total), ruff clean.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* docs(eventing): record T5 as implemented, and what it does not yet give you

The status lines said steps 5-12 were design only; T5 (per-user stores and the global
ownership index) is now in the tree, so they were wrong in the direction that matters —
a reader would either re-implement it or, worse, read "per-user stores" as "isolation".

The distinction the docs now draw explicitly: each tenant's sessions, responses and
transcripts DO live in their own SQLite files, and a read routes to the owning tenant's
store — but nothing yet checks that the *caller* is that owner, because owner-scoped
reads (T7) and transcript auth (T8) are not implemented. So `multi` mode's reads,
`/continue` and `PUT /transcript` remain as open as Phase 2's.

Also adds `EB_MAX_OPEN_STORES` to the component README's Phase 3 table with the
descriptor arithmetic that explains the default, and to `test_manifests.py`'s
classification list so T22's "every Phase 3 variable is on one side of the
Secret-vs-ConfigMap rule" check keeps covering it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): address the five must-fixes from the #883 review

Every finding reproduced against 16158e5 before fixing, and every fix has a regression
test in `tests/test_review_fixes.py` that fails without it (verified by mutation, one
fix at a time). They are filed together because they share a cause worth naming: all
five were invisible to 192 test functions, because the fixtures assembled the
collaborators by hand with the arguments the production call sites fail to supply.

**1. `single`-mode regression: the ownership index was never seeded.** The serious one,
and against the exact guarantee this phase claims. Dropping Phase 2's `Minter` seeding
loop left nothing to populate its replacement, so on the first start after an upgrade
`owners.sqlite` is empty while `sessions.sqlite` is full and `exists()` reports live ids
as free. With ~1,000 existing correlations a mint has roughly 1-in-25,000 odds of
reissuing one, and then `upsert_session` overwrites a session and the new prompt appends
to somebody's existing conversation. Phase 2's odds were zero.

`OwnerIndex.seed_from()` plus a one-time backfill from every store at startup.
§2.6 is right that per-start seeding is the wrong trade; it accidentally argued away the
one-time backfill too, and §6.1a now records the distinction.

**2. `create_group` did not pass the userkey.** `submit_members` got it and
`groups.create()` did not, so in `multi` the group row and the groupid's owner claim
landed in the shared store while every member row landed in the tenant's: 0 members
forever, `maybe_complete` never fires, the batch never completes and never notifies. A
retried POST with an Idempotency-Key read the tenant store `create` never wrote and
returned `members: []` — worse than a second batch, because it looks like success.

**3. `close_group`/`cancel_group` were unauthenticated cross-tenant mutations.** Neither
called `_caller`; both resolved any tenant's group from the global index and acted on it.
T7 covers *reads* staying as open as Phase 2's — Phase 2 had no tenants to cross and no
cross-tenant mutation anywhere, so this was a capability being introduced, not an openness
preserved. Now gated by `_authorize_mutation`, answering `404` for another tenant's group
per §6.2's enumeration rule. Single-tenant mode is untouched.

**4. `agent` was outside `SIGNED_ATTRS`** while `ce_agent` selects the AgentSpec that
supplies `--permission-mode` and the tool allowlists — §5.1's "sandbox". A forged value
widened the sandbox and the signature still verified, making a signed deployment worse
than an unsigned one because an operator believes it is attested. Added in the same
change as `userkey`/`depth`, per §8.3's one-canonicalisation-break rule.

**5. `userkey` reached a filesystem path join unvalidated**, from an inbound Kafka
header, and `Store.__init__` calls `mkdir(parents=True)` — so `..` created and wrote to a
directory inside another tenant's store, or outside the tree. `tenancy.is_valid_userkey()`
is the gate, exported from the module that is "the ONLY place an identity becomes a name"
so a consumer can check the shape the producer guarantees. An invalid key is treated as a
missing one — `shared/` plus `unattributed` — so a forgery is counted and visible rather
than acted on. Note it is *truthy*, so the pre-existing `not userkey` check let it past.

Also from the review:

- **A known-but-unowned correlation 500'd on `/continue`.** `owner=None,
  unresolved=None` is falsy, so the handler proceeded, wrote a prompt row, then
  `TopicSet.requests(None)` raised — a traceback with an orphan row already committed, so
  the conversation showed a turn that was never submitted. Reading and publishing have
  different requirements, so `_publishable_owner` is now separate from `_owner_userkey`:
  shared-tier data stays readable, and a turn for it is refused before anything is written.
- `claim()` uses `ON CONFLICT DO NOTHING` + a follow-up read, so `Collision` is the only
  way a conflict presents whether or not two writers race the in-process lock — the bare
  `INSERT` would have raised `IntegrityError`, which `Minter.mint` deliberately does not
  catch.
- `created_utc` at millisecond precision (matching `ce.now_iso()`) with a `rowid`
  tiebreaker; at second granularity a 100-member batch had arbitrary order among its rows.
- A bad `agent` name is a `400` on the call that made the mistake, not a `202` followed by
  an async `phase=error` the submitter may never read.
- `max_turns: 0` is no longer silently rewritten to the spec's value while the spec path
  rejects zero.
- `EB_TOPIC_PREFIX` validated (it is the one topic-name component `_slug` never sees, and
  `.`/`_` there reproduce the JMX metric collision) and `EB_MAX_OPEN_STORES` refuses
  readably instead of raising a bare `ValueError`.
- Dropped a flaky assertion: `"gh" not in` 26 characters of base32 fails ~2.4% of the
  time by construction, and the `"mrsabath" not in` line above it is the real property.
- `_canonicalise`'s docstring no longer claims static identities are byte-exact when the
  `@` branch catches address-shaped ones first.

Design doc, where the code had disproved it (§9's "the code is the authority" rule):
§6.1's layout corrected to put single-tenant stores in the bridge root; "Closing is safe"
replaced with what commit 49aed05 established, including that the first fix attempt only
pinned the SSE path and left the class open; new §6.1a documents `owner_index.py`; §2.6's
"one new extension attribute" corrected to four, with which three are signed and why; §3.1
records the single-mode `events`/`dead` names. The status header now warns that `multi`
does not work end to end until T6 — no response is consumed, which is more useful than
"not yet isolated".

Tests: 45 new (840 total), ruff clean.

Still outstanding from the review and deliberately not in this commit: the T22 subset
check, `RequestsMirror`/`NtfyPublisher` `stores=`, the handler-to-topic integration test,
the `forget()` tombstone decision, and the four documents.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): the four remaining #883 suggestions, and tombstone deletion

Round two of the review, covering what was deferred from 6ebdd80.

**T22 now checks the direction that enforces the rule.** `CONFIGMAP_VARS ⊆ found`
catches a stale entry and nothing else; the direction that catches a *new* variable
classified as neither is `found ⊆ CONFIGMAP_VARS ∪ SECRET_PATH_VARS`, which is the case
the test exists for. Scoped to Phase 3's variables — the 36 that pre-date §8.3 are listed
explicitly, with `test_the_baseline_list_is_still_accurate` keeping that list honest
rather than letting it silently widen the exemption.

Verified by deliberate failure, as asked: injecting an unclassified `e("EB_UNCLASSIFIED_X")`
read into `config.py` turns the test red with the variable named. Also replaced the
hardcoded ten-space indentation in the inlined-secret check with `\s*`, matching the
precedent elsewhere in the same file — the old pattern would miss the same variable
inlined in an initContainer, a sidecar or a deeper patch.

**`RequestsMirror` and `NtfyPublisher` now take `stores=`.** Both knew the per-tenant
topic names and still wrote to the shared store, which does look like the sweep in
77807b6 not reaching them. The mirror back-filled every prompt into `shared/`, so a
correlation whose request the bridge did not originate showed a BLANK PROMPT on its
owner's page — the exact gap the mirror exists to close. ntfy's `_last_prompt` and
`_last_assistant_text` read only the shared store, so a tenant's notification arrived
without the prompt or the reply, which §4.1 says is the content that lets a phone that
cannot reach EventBridge still see the result. Its group-existence check is routed too.

**A handler→topic integration test**, which the review argued was a better investment
than more unit tests on the pieces. It is: faking at `KafkaProducer` rather than at
`Producer` means the real handler, the real `Producer` and the real `TopicSet` all run,
and the assertion is *which topic the bytes went to*. Confirmed by reverting each fix that
it catches both defects it was written for — the `create_group` missing `userkey` and the
`/continue` 500 on an unowned correlation.

That test immediately earned itself twice over: the mutation run surfaced that
`_publish_started`/`maybe_complete` swallow `ValueError` from `TopicSet` into a log line,
so a caller forgetting a `userkey` produced a group that silently never announced itself —
the same shape as the `TypeError` already guarded there. Both guards now cover both.

**Deletion tombstones rather than freeing the id.** This is the one I said needed thought
rather than a quick answer, and the answer is that reuse is unsafe. Verified: `forget()`
used to `DELETE` the row, `exists()` is the `Minter`'s only uniqueness check, and
`sessionuuid = uuid5(NAMESPACE, correlationid)` is unsalted by design — so a reissued
`correlationid` derives *the same session uuid*, and a `claude` transcript left on a
runner's volume becomes resumable by the new correlation. That is one user's conversation
continuing inside somebody else's agent.

So `forget()`/`forget_tenant()` keep the row, set `deleted_utc` and clear `userkey` — the
id is reserved forever, and the tombstone discloses nothing about whose it was. The
alternative is guaranteeing every transcript keyed on that uuid is purged everywhere,
including volumes this process does not own, which is not a guarantee EventBridge can
make. Cost is one short row per deleted correlation. Migrated with an `ALTER TABLE`, the
same idiom `store.py` uses for `prompts.submitter`, and a test covers an index created
before the column existed.

The reviewer was also right that the docstring contradicted the code: it claimed "freeing
the id for reuse is deliberate" about a `DELETE` nobody had reasoned about.

Design doc: §6.1a records the tombstone decision and why, since the index is what
enforces it.

Tests: 855 passing (+13), ruff clean.

Still outstanding: the four phase documents (`IMPLEMENTATION_REPORT3.md`,
`README_PHASE3.md`, and the two owed Phase 2 documents).

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* docs(eventing): IMPLEMENTATION_REPORT3.md — what Phase 3 built, measured

Asked for in the #883 review, and the argument for writing it now rather than later was
the most persuasive one there: the findings are attributable to specific commits today and
will not be in a month. 361 lines, following `IMPLEMENTATION_REPORT1.md`'s contract —
measured figures rather than estimates, failures with root causes, and an explicit account
of what is NOT verified.

Phase 0 and Phase 1 each ship three documents; Phase 2 shipped none, and the gap shows in
that there is nowhere to put the §6.1 split-consumer trap except inside a design document.
This starts closing that for Phase 3.

**Measured rather than asserted.** §6.1 estimates the per-tenant descriptor cost and
derives a ceiling; measured it is 6.0 descriptors per open store (10 tenants, 60 fds),
which puts ~680 tenants under this host's 4096 soft limit and ~170 under the 1024 the
design assumed — so squarely inside its "150-200" once the limit is held constant. The
ownership index costs 27.4 µs per `claim`, 1.7 µs per `exists`, and 270 ms to seed 10,000
ids. That last figure is the one that matters: it settles §2.6's objection to seeding,
because the *one-time* backfill it appeared to rule out costs 270 ms once, while the
*per-start* seeding it correctly rejects is what the index exists to avoid.

**Seven deltas from the design, recorded as decisions rather than drift** — the
single-tenant store location, eviction by reference-drop, tombstoned deletion, the
single-mode `events`/`dead` names, `agent` joining `SIGNED_ATTRS`, the exported userkey
validator, and prefix validation.

**The findings, ordered by what they cost to find.** Three are worth the document on their
own:

- The `single`-mode regression shipped behind a headline claiming it could not happen,
  with the reproduction (50 of 50 live ids reported free), the 1-in-25,000 arithmetic over
  a verified 50x50x10,000 id space, and the three reasons it is instructive: it was in the
  default configuration, the suite could not see it because no fixture simulated an
  upgrade, and §2.6's wording contributed.
- The eviction contract, where the lesson is the *first* fix: pinning only the SSE path
  addressed the case that was easy to picture and left the class open.
- `except Exception` converting three separate programming errors into log lines, on three
  separate occasions, which is why it is filed as one finding with the pattern extracted.

Also records the suite's structural blind spot as a finding in its own right — a fake
placed at the boundary under test cannot test that boundary — since that is what let two
defects survive 200-odd new test functions.

**§5 is the longest section, deliberately.** `multi` does not work end to end until T6 (no
response is consumed); no cluster, no two-tenant deployment, no Kafka; reads not
owner-scoped; Tier A is not isolation; the per-tenant throughput ceiling §10 asks for is
still unmeasured; the `claude --help` flag assertion does not run in CI; `seed_from` is
capped at 10,000 per store; and `is_valid_userkey` is shape validation, not authorisation.

Every figure in the report was measured on this branch, every cited commit verified to be
on it, and every cited design section verified to exist — §21.6 turned out to be
`DESIGN_PHASE1.md`'s and is now qualified as such. Two numbers in my own PR description
were wrong and are corrected here: the test delta is +232 unique functions (525 to 757),
not the 139 or 217 claimed earlier.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* fix(eventing): two orphan-write paths, and three design/code gaps (#883 round 2)

Both must-fixes reproduced against c25a0c7 before fixing, and both are my own round-1
fix introducing a new bug class: `6ebdd80` added validation of the request's `agent`
name at the HTTP boundary — right place, right reason — and placed it AFTER the writes.

**create_group** had already committed the group row and published `group.started` when
the 400 returned, and because the row is keyed on the Idempotency-Key the state was
permanent: every retry read back `created: false, members: []`, which looks like success.
That is verbatim the failure the `userkey=` comment added in the same round describes,
reached by a route that fix did not cover.

    POST /v0/groups {"prompts":["a","b"],"agent":"../../etc"}  Idempotency-Key: retry-me
      -> 400,  topics: ['kev1-st-alice-c8ff0431-responses'],  groups: ['wild-weasel-1995']
      retry, same key, valid name -> 200 {created: false, members: []}

**start_agent** had written the session row, the prompt row and claimed the correlation
id — a transcript showing a turn that was never submitted, plus an id nothing ever
un-claims, since tombstoning is for deletion rather than abandonment.

Both resolutions now happen before the first write. `_agent_for` needs only `caller` and
`body`, so there was never anything to gain by deferring it. After the fix both paths
publish nothing, write nothing, and the group retry launches the batch properly.

The reviewer's observation about the tests is the more useful half: the assertion that
would have caught both — `get_prompts(corr) == []`, *a refusal writes nothing* — already
existed 25 lines earlier in the same file, on a different route. The new tests assert it
on both paths, with the Idempotency-Key retry for the group case, and were verified by
reverting each ordering (7 failures).

### Three design/code gaps §3 of the report had missed

**The backfill was per-start, not one-time.** §6.1a says "seeded once" and the report
said "once, on the first start after an upgrade"; the loop re-opened every tenant store
and re-read `all_correlations` on every boot — exactly the N-store startup seeding §2.6
rejects, reintroduced by the fix for the missing seed. `INSERT OR IGNORE` kept it correct;
what repeated was the cost §2.6's argument is entirely about. A `seed_state(scope,
done_utc)` table now records each scope, and `needs_seeding` is checked before the store
is opened. Measured at 100 tenants x 200 correlations:

    before:  start 1  702 ms / 101 opens    start 2  210 ms / 101 opens
    after:   start 1  702 ms / 101 opens    start 2  0.8 ms /   1 open

Sound because nothing can add an unindexed id to a seeded store: every mint claims as it
mints.

**`AgentSpec.limits` was parsed and enforced nowhere, silently.** `Skill` states plainly
that it carries `sha256` for T19; `Limits` made no such disclaimer, so `timeout_s = 900`
loaded clean and did nothing. Now says so, with which field belongs to which later task.
Its three conversions also escaped the module's own `SpecError` contract —
`timeout_s = "900s"` raised a bare `ValueError` past `run_agent`'s handler instead of
becoming the `phase=error` event §5.1 promises.

**§8.3 disagreed with §2.6 and the code.** It still said the signed set "changes twice"
and named two attributes, while §2.6, `signing.py` and `ce.py` all said three — and
`signing.py` cited §8.3 as the authority for a rule §8.3 stated over the wrong count.

Two smaller ones from the same pass: a tombstoned correlation could be re-claimed by the
NULL owner (`row[0] == userkey` reads as "already mine" when both are NULL),
contradicting what `forget` guarantees — unreachable from `Minter.mint`, which checks
`exists()` first, but reachable as soon as a correlationid arrives from outside, which
§7's triggers will bring. And `registry.parse` accepted `agent` unchecked, the only field
it did not validate, in a function whose docstring promises every inconsistency is a
refusal; a typo there meant an asynchronous `phase=error` on every request from that user
rather than a startup refusal. Verified: the registry now refuses with the bad value named.

### Documentation

- The component README's Phase 3 block no longer orphans the "Two of these are
  load-bearing" sentence from its payoff 25 lines later — the block moved below it under
  its own `#### Phase 3` heading, and the note count is now right (three, not two). Added
  the T6 "does not work end to end" warning here too.
- `IMPLEMENTATION_REPORT3.md` §1 now counts from a named point (`3278d75`) instead of
  "currently", since a report cannot be right about the diff of the commit containing it;
  both figures are given. New §4.7 and §4.8 record this round's findings, including that
  fixing one orphan-write instance did not teach me to look for the class — the same
  lesson §4.2 records about the eviction contract, learned a second time.

Tests: 860 passing (+5), ruff clean, both multi-mode startup paths smoke-tested.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* test(eventing): Phase 3 — regression tests for the three untested fixes (#883)

From the #883 approval: reverting `agentspec`'s `try/except`, `claim`'s tombstone check
or `registry`'s `validate_name` left all 858 tests passing, so three of the six fixes in
`258da7a` were invisible to the suite. The standard this PR has applied since `6ebdd80`
is "every fix has a regression test, verified by reverting its fix", and it had been
applied to half that commit — and, as the review put it, the three without one are the
three whose absence is invisible.

Fourteen tests, each verified by reverting its fix independently:

| Fix | Failures when reverted |
|---|---|
| `[limits]` conversions wrapped in `SpecError` | 4 |
| tombstone never re-claimable | 1 |
| `registry.parse` validates `agent` | 5 |

The tombstone test covers the case that was actually wrong — `claim(corr, None)` against
a tombstoned row, where `row[0] == userkey` is `None == None` and read as "already
mine" — and also asserts a *tenant* re-claim now fails with "tombstoned" rather than
"already owned by (single-tenant)", so the message points at the real reason.

The registry tests cover the shapes `validate_name` rejects and, separately, that a
valid name loads and an absent one stays empty so `ER_AGENT_NAME` still applies. A gate
that rejects what it should allow is the other way to get this wrong.

**Also closes the range-check gap the review flagged as on-the-record.** `timeout_s = -5`
and `max_events = 0` loaded clean:

    timeout_s = -5.0 | max_events = 0

Defensible while `Limits` is documented inert, but a negative deadline is nonsensical
input whether or not anything reads it yet, the module's own standard is that a policy
which cannot be understood fails to load, and the validation was already right there.
Rejecting now also means whoever wires up the `Popen` wrapper inherits a value they can
trust instead of re-validating. `0` keeps its documented meaning for `timeout_s` and
`max_output_bytes` ("no deadline" / "unbounded"), with a test pinning that the range
check does not reject the sentinels; `max_events = 0` is refused because a run that may
emit no events cannot report its own result.

Tests: 875 passing (+14), ruff clean.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

* test(eventing): extend the §8.3 rule to #884's seed paths, and cover kind-signed

Follow-up to rebasing onto `main` now that #884 and #890 have merged. Both predicted
conflicts were in `agentdocs/README.md` and resolved as planned — `main`'s Phase 2 row
(which gained §8 from #890) plus this branch's Phase 3 row. `test_manifests.py`
auto-merged, keeping #884's `allowed_secrets` fix beside this branch's Phase 3
classification.

That left two things the merge did not resolve by itself.

**Extending `SECRET_PATH_VARS` to #884's seed paths needed the rule split, not widened.**
#884's review predicted this tuple would be extended to `ER_/EB_SIGNING_KEY_PATH` and
checked that doing so stays green. It does for the ConfigMap half and would NOT for the
literal-value half, because those two variables hold a **mount point, not key material** —
and #884 deliberately sets them as container `env` entries so the path sits beside the
volume that supplies the seed, which is what `configmap.yaml` asks for.

So the tuple is now two:

- `SECRET_VALUE_VARS` — the three whose *value* is key material. No literal value
  anywhere, which is the original assertion.
- `SEED_PATH_VARS` — #884's two. A literal value is correct; a ConfigMap key is not.

Both still obey "never from `config.toml`", so that check covers all five now where it
covered three. Treating a path naming a secret as if it were the secret would have forced
#884 to either revert its own correct fix or carve itself out of the rule — the distinction
is the thing worth encoding.

**`kind-signed` was not in `OVERLAYS`.** #884 added the overlay and every assertion in
this file silently skipped it — including the seed-path rule the PR exists to respect, the
`replicas` invariants, and the no-credential checks. Added, which is also what makes the
new assertion meaningful: with `kind-signed` absent, injecting a seed path back into
`eventing-config` passed; with it present, that regression fails as it should. Verified
both ways.

Tests: 899 passing (875 here + #884's 24 picked up in the rebase), ruff clean, all four
overlays build.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>

---------

Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants