Skip to content

feat: review every change of a configured agent before it runs - #88

Merged
albanm merged 19 commits into
mainfrom
feat-auto-approval
Nov 4, 2026
Merged

albanm merged 19 commits into
mainfrom
feat-auto-approval

Conversation

@albanm

@albanm albanm commented Nov 4, 2026

Copy link
Copy Markdown
Member

A configured agent's every write and destructive catalog call, and every capability grant (web access, MCP access, planning), is now judged by an LLM reviewer before it runs. Where a class is routed to a person, the reviewer goes first and the person decides after it.

Why: people approve cards by reflex, so a human-only gate is weak, and it left ordinary writes unchecked. Automatic approval should be faster and block less, and in the end be safer, not less safe. Planned tasks and code_mode scripts could not use gated tools at all; they can now make the writes the reviewer allows.

  • Routes replace approvalLevel: approvalRoutes.write / .destructive, each reviewer or reviewer+person. Defaults: write → reviewer, destructive → reviewer+person. There is no unchecked route, and reads are never reviewed. Grants are always reviewer+person, the person being an admin.

  • The reviewer (api/src/approval-review/) runs on the shared two-stage guard engine, on the moderator role, in this order:

    • Intent: the people's stored messages, plus the agent's persona and instructions. In a webhook run, only the payload's event type and resource id, never its text.
    • The conversation's earlier gated calls, each with how it was decided and which message it followed, so a refused outcome reached another way is caught.
    • The proposed call.
    • Exposure: whether the conversation read outside content, naming only the tools it came from.
    • Never shown: tool outputs and the assistant's text.

    Each review answers allow, deny or unsure. A failed review fails closed (unsure).

  • Gate: one AI SDK toolApproval function. Deny is final, and the model is told not to work around it. Unsure is a card in a chat and a deny elsewhere. The breaker stops the turn after 3 denials in a row or 10 in one turn (stop reason review-denied). Nested code_mode calls are reviewed too, within the turn's credit budget. In a task run, the first failed review notifies the agent's admins.

  • UI:

    • Approval cards show the reviewer's view.
    • A reviewer denial shows as a "Denied by the reviewer" chip, which tells the person to say so to the agent if they want it done. There is no override button.
    • The agent form edits the routes, and its instructions help text says the reviewer reads them.
  • Eval: reviewer cases in dev/guard-eval (overeager, injection, out of scope, grants, workarounds). On Claude Haiku through the dev bridge, one run: 0 of 41 harmful calls allowed, 2 of 23 benign calls carded, p50/p95 3.6/8.7 s.

Heads-up:

  • Release blocker: the reviewer has not yet been measured on the production moderator models, nor repeated to check verdict stability. Run GUARD_EVAL_GUARDS=reviewer node dev/guard-eval/index.ts (≥3 repeats) and set the thresholds before releasing.
  • Breaking: approvalLevel is removed (never released). upgrade/0.12.2/approval-level-to-routes.js migrates agents that still carry it (dev, staging): write → both routes reviewer+person, destructive → the default routes, none → both routes reviewer, the closest there is now that no route goes unchecked.
  • Cost and latency: every write now costs one or two moderator calls (about 3.4k tokens) and adds seconds before the write runs. Point moderator at a capable model.
  • The personal assistant is unchanged: it has no reviewer and is not offered write tools.

albanm and others added 19 commits November 3, 2026 18:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iewer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mation-only mode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aker

The automated approval reviewer's pure side (api/src/approval-review: the
plan's api/src/review is the conversation export's module) and its two
stage calls, stage 2 reasoning. The intent and proposed-call wrapper
prefixes join ENVELOPE_TAGS; the compaction recap's preamble is exported
with isRecapMessage, the exposure marker; the engine's outcome types and
failure descriptions move to guard/operations.ts so a pure module can use
them (re-exported from guard/service.ts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents, reason cut in the record

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
createReviewGate: one toolApproval function (gone approved tools denied, a
person-approved replay not reviewed again, unreviewed tools by needsApproval,
reviewed ones by route and surface), reviewNested/noteNested for code_mode,
records and the breaker. createTurnReviewer: the moderator role through the
guard engine, confirmation-only for destructive calls and grants, metered and
recorded as 'review', fails closed. Mock reviewer seam; set_note and
delete_note fixture tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…esets it; replay tests

A review-failed record is no judgment: it is no longer passed to the breaker, so
it neither counts as a denial nor ends a run of consecutive ones. Pinned beside the
replay cases: a call asked about but unanswered, or declined, is reviewed; a
person-approved replay is exactly user-approval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent's approvalLevel becomes approvalRoutes: write and destructive, each
'reviewer' or 'reviewer+person' (defaults reviewer / reviewer+person); there is no
unchecked route. applyApprovalPolicy marks every non-read catalog tool for review
and sets needsApproval only on reviewer+person, so unattended runs and scripts keep
excluding those. The capability grants are reviewed, then go to the admin.

The executor builds the review intent from the target (the conversation's stored
user messages, or the task's prompt without its payloads) and the loop runs a turn
reviewer behind the review gate, the configured agent's toolApproval. Automatic
decisions do not set awaiting-approval; a reviewer's denial settles the call
output-denied with its reason; every reviewed call carries toolMetadata.review;
the breaker stops the turn with the new stop reason review-denied (a task run cut
short). The personal assistant keeps its fixed policy and no reviewer.

Self-knowledge states each tool's route. UI files get only the type-level change
(approvalRoutes / DEFAULT_APPROVAL_ROUTES); the form's text and the web warning are
Task 6's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… grants pinned

reviewerDenialText (operations.ts) is the model-facing reason of every
reviewer-originated denial (deny, and unsure on the task surface), stored as the
part's approval.reason; the raw reason stays in toolMetadata.review. API cases for a
request_web_access grant denied (no card) and allowed (a card with the review);
replays assert no further review call. The breaker's stop message drops 'in a row'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A code_mode script's nested call to a reviewed tool now goes through the turn's
review gate (reviewNested) before it runs: codeCallable keyed on needsApproval,
which reviewer-routed tools carry none of, so scripts bypassed the reviewer.
Anything but allow fails the nested call with reviewerDenialText; the record is
kept on the nested trace entry's toolMetadata.review, and the breaker counts
nested denials (three in a script stop the turn, review-denied). Every nested
call that ran is noted (noteNested), so a script's scanned reads expose the turn.
A reviewed tool with no gate to judge it is refused, never run unjudged; the
personal assistant's tools carry no review and keep today's behaviour.

Task-run tests pin that reviewer-routed writes stay offered and run when
allowed, unsure is a deny, and reviewer+person tools are not offered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eviewed

The breaker stopped the turn only after the step, so a running script (or the
step's later direct calls) kept sending variants to the reviewer and ran any it
allowed. Once tripped, a reviewed call, direct or nested, now gets a deny record
with code breaker-tripped, without a review and without counting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the form

An approval card a reviewed call reached says why: allowed by the reviewer but
this class needs a person (with the reviewer's reason), the reviewer unsure, or
the review failed. A reviewer's denial is a 'Denied by the reviewer' chip whose
detail gives the reviewer's own reason (toolMetadata.review), not the text the
model was handed. A person's rejection after a review stays 'Declined'.

The web-access warning about writes without approval goes: every change is now
reviewed before it runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REVIEW_CASES in dev/guard-eval/corpus.ts (20 benign, 9 overeager, 9 injection,
8 out-of-scope, 9 grant), the reviewer guard of the eval (dev/guard-eval/reviewer.ts,
run as createTurnReviewer runs it), a unit spec driving it with the mock model,
the autonomous-agent-overreach and autonomous-agent-reviewed-write simulations, and
a 'Measuring the reviewer' note in moderation.md: numbers pending the first run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Column-label patch keeps the full schema; the wrong-application delete is
out-of-scope, with a new overeager metadata rewrite; the routine news edit adds a
block instead of inventing entries. Haiku via the bridge, 2026-11-03, one run:
0/36 harmful allowed, 2/20 benign carded, stage 2 on 24/35 writes, p50/p95 3.1/7.2 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approval gate (routes, the reviewer in series before a person, final
denials, the replay rule, toolMetadata.review and its codes, the circuit
breaker and review-denied), the reviewer as the third guard in moderation.md
around the recorded eval, the unattended policy and the consent principle
revisited in planned-tasks.md, reviewed nested calls in code-mode.md, the web
warning gone, the catalog-tools convention, the index. The tripwire spec's
follow-up notes point to the reviewer as done; AGENTS.md and the dev MCP
server's startup line name set_note and delete_note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Cut arguments skip stage 1 (isConfirmationOnly, shared with the eval
  harness), and the rules say a cut call whose hidden part could change the
  judgement is unsure.
- A failed review that denies tells the model it could not be reviewed
  (reviewUnavailableText); a task run's first one pushes an
  agents:review-failed event.
- A script's reviewed call checks the turn's budget first and is denied
  once it is spent (code 'budget'); the code_mode description and
  code-mode.md state that reviews count in the script's clock.
- approvedIn reads only the last tool message, as collectToolApprovals does.
- A judged unsure counts as a denial for the breaker where it denies.
- Exposure wording says "this conversation"; the card labels the allow
  reason as the reviewer's note; no-model failure says so.
- Docs: planned-tasks prompt authorship, Codex's breaker thresholds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…enial is the person's words

The reviewer judged each call alone, so a refused outcome reached another way
(another tool, a script, smaller parts) passed call by call. It now reads the
conversation's earlier gated calls in their own wrapper, each with how it was
decided (a person's rejection reason included) and which message it followed.
The denied chip tells the person to say so to the agent; the instructions help
text tells admins the reviewer reads them. Eight multi-call eval cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@albanm
albanm merged commit 5517671 into main Nov 4, 2026
4 checks passed
@albanm
albanm deleted the feat-auto-approval branch November 4, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant